Python - Looping through API and writing Dict to csv
up vote
0
down vote
favorite
I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?
See my code here:
ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba',
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'
for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]
try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)
json_obj = urlopen(URL)
data = json.load(json_obj)
with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())
time.sleep(1)
Sample output:
data Out[59]:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
python api csv web-scraping
add a comment |
up vote
0
down vote
favorite
I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?
See my code here:
ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba',
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'
for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]
try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)
json_obj = urlopen(URL)
data = json.load(json_obj)
with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())
time.sleep(1)
Sample output:
data Out[59]:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
python api csv web-scraping
What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26
I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28
Your first problem is a trivial one, you just need to addlineterminator='n'as a keyword argument in yourcsv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
– Idlehands
Nov 19 at 15:29
@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42
@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?
See my code here:
ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba',
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'
for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]
try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)
json_obj = urlopen(URL)
data = json.load(json_obj)
with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())
time.sleep(1)
Sample output:
data Out[59]:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
python api csv web-scraping
I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?
See my code here:
ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba',
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'
for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]
try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)
json_obj = urlopen(URL)
data = json.load(json_obj)
with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())
time.sleep(1)
Sample output:
data Out[59]:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
python api csv web-scraping
python api csv web-scraping
edited Nov 19 at 15:49
Idlehands
3,5701417
3,5701417
asked Nov 19 at 15:13
Carolin
1159
1159
What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26
I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28
Your first problem is a trivial one, you just need to addlineterminator='n'as a keyword argument in yourcsv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
– Idlehands
Nov 19 at 15:29
@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42
@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44
add a comment |
What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26
I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28
Your first problem is a trivial one, you just need to addlineterminator='n'as a keyword argument in yourcsv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
– Idlehands
Nov 19 at 15:29
@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42
@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44
What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26
What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26
I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28
I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28
Your first problem is a trivial one, you just need to add
lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.– Idlehands
Nov 19 at 15:29
Your first problem is a trivial one, you just need to add
lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.– Idlehands
Nov 19 at 15:29
@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42
@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42
@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44
@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:
w = csv.writer(f, delimiter=";", lineterminator='n')
For your second problem, you can flatten your json before passing into csv:
for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v
This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.
Result:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
Now you can just pass that into your csv.writer().
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:
w = csv.writer(f, delimiter=";", lineterminator='n')
For your second problem, you can flatten your json before passing into csv:
for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v
This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.
Result:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
Now you can just pass that into your csv.writer().
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
add a comment |
up vote
0
down vote
accepted
As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:
w = csv.writer(f, delimiter=";", lineterminator='n')
For your second problem, you can flatten your json before passing into csv:
for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v
This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.
Result:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
Now you can just pass that into your csv.writer().
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:
w = csv.writer(f, delimiter=";", lineterminator='n')
For your second problem, you can flatten your json before passing into csv:
for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v
This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.
Result:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
Now you can just pass that into your csv.writer().
As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:
w = csv.writer(f, delimiter=";", lineterminator='n')
For your second problem, you can flatten your json before passing into csv:
for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v
This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.
Result:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
Now you can just pass that into your csv.writer().
answered Nov 19 at 15:47
Idlehands
3,5701417
3,5701417
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
add a comment |
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377573%2fpython-looping-through-api-and-writing-dict-to-csv%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26
I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28
Your first problem is a trivial one, you just need to add
lineterminator='n'as a keyword argument in yourcsv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.– Idlehands
Nov 19 at 15:29
@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42
@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44