Python - Looping through API and writing Dict to csv











up vote
0
down vote

favorite












I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?



See my code here:



ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba', 
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'

for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]

try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)

json_obj = urlopen(URL)
data = json.load(json_obj)

with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())

time.sleep(1)


Sample output:



data Out[59]: 
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}









share|improve this question
























  • What does your program print out, if anything? Could we get an example response JSON?
    – Aquarthur
    Nov 19 at 15:26










  • I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
    – Idlehands
    Nov 19 at 15:28












  • Your first problem is a trivial one, you just need to add lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
    – Idlehands
    Nov 19 at 15:29










  • @Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
    – Carolin
    Nov 19 at 15:42












  • @Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
    – Carolin
    Nov 19 at 15:44















up vote
0
down vote

favorite












I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?



See my code here:



ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba', 
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'

for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]

try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)

json_obj = urlopen(URL)
data = json.load(json_obj)

with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())

time.sleep(1)


Sample output:



data Out[59]: 
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}









share|improve this question
























  • What does your program print out, if anything? Could we get an example response JSON?
    – Aquarthur
    Nov 19 at 15:26










  • I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
    – Idlehands
    Nov 19 at 15:28












  • Your first problem is a trivial one, you just need to add lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
    – Idlehands
    Nov 19 at 15:29










  • @Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
    – Carolin
    Nov 19 at 15:42












  • @Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
    – Carolin
    Nov 19 at 15:44













up vote
0
down vote

favorite









up vote
0
down vote

favorite











I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?



See my code here:



ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba', 
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'

for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]

try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)

json_obj = urlopen(URL)
data = json.load(json_obj)

with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())

time.sleep(1)


Sample output:



data Out[59]: 
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}









share|improve this question















I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?



See my code here:



ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba', 
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'

for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]

try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)

json_obj = urlopen(URL)
data = json.load(json_obj)

with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())

time.sleep(1)


Sample output:



data Out[59]: 
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}






python api csv web-scraping






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 19 at 15:49









Idlehands

3,5701417




3,5701417










asked Nov 19 at 15:13









Carolin

1159




1159












  • What does your program print out, if anything? Could we get an example response JSON?
    – Aquarthur
    Nov 19 at 15:26










  • I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
    – Idlehands
    Nov 19 at 15:28












  • Your first problem is a trivial one, you just need to add lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
    – Idlehands
    Nov 19 at 15:29










  • @Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
    – Carolin
    Nov 19 at 15:42












  • @Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
    – Carolin
    Nov 19 at 15:44


















  • What does your program print out, if anything? Could we get an example response JSON?
    – Aquarthur
    Nov 19 at 15:26










  • I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
    – Idlehands
    Nov 19 at 15:28












  • Your first problem is a trivial one, you just need to add lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
    – Idlehands
    Nov 19 at 15:29










  • @Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
    – Carolin
    Nov 19 at 15:42












  • @Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
    – Carolin
    Nov 19 at 15:44
















What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26




What does your program print out, if anything? Could we get an example response JSON?
– Aquarthur
Nov 19 at 15:26












I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28






I'm not exactly sure what the site is, but it might not be wise to share your token information publicly.
– Idlehands
Nov 19 at 15:28














Your first problem is a trivial one, you just need to add lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
– Idlehands
Nov 19 at 15:29




Your first problem is a trivial one, you just need to add lineterminator='n' as a keyword argument in your csv.writer(). Your second one is less trivial and will be helpful to see some sample structure of the data.
– Idlehands
Nov 19 at 15:29












@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42






@Idlehands: Your suggestion to the first problem worked, thank you! I uploaded a sample output above
– Carolin
Nov 19 at 15:42














@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44




@Idlehands: It is a public API with no personal info, so no problem here. Just like any other movie title example ;-)
– Carolin
Nov 19 at 15:44












1 Answer
1






active

oldest

votes

















up vote
0
down vote



accepted










As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:



w = csv.writer(f, delimiter=";", lineterminator='n')


For your second problem, you can flatten your json before passing into csv:



for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v


This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.



Result:



{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}


Now you can just pass that into your csv.writer().






share|improve this answer





















  • Thank you, works as intended :-)
    – Carolin
    Nov 19 at 16:06











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377573%2fpython-looping-through-api-and-writing-dict-to-csv%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote



accepted










As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:



w = csv.writer(f, delimiter=";", lineterminator='n')


For your second problem, you can flatten your json before passing into csv:



for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v


This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.



Result:



{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}


Now you can just pass that into your csv.writer().






share|improve this answer





















  • Thank you, works as intended :-)
    – Carolin
    Nov 19 at 16:06















up vote
0
down vote



accepted










As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:



w = csv.writer(f, delimiter=";", lineterminator='n')


For your second problem, you can flatten your json before passing into csv:



for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v


This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.



Result:



{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}


Now you can just pass that into your csv.writer().






share|improve this answer





















  • Thank you, works as intended :-)
    – Carolin
    Nov 19 at 16:06













up vote
0
down vote



accepted







up vote
0
down vote



accepted






As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:



w = csv.writer(f, delimiter=";", lineterminator='n')


For your second problem, you can flatten your json before passing into csv:



for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v


This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.



Result:



{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}


Now you can just pass that into your csv.writer().






share|improve this answer












As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:



w = csv.writer(f, delimiter=";", lineterminator='n')


For your second problem, you can flatten your json before passing into csv:



for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v


This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.



Result:



{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}


Now you can just pass that into your csv.writer().







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 19 at 15:47









Idlehands

3,5701417




3,5701417












  • Thank you, works as intended :-)
    – Carolin
    Nov 19 at 16:06


















  • Thank you, works as intended :-)
    – Carolin
    Nov 19 at 16:06
















Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06




Thank you, works as intended :-)
– Carolin
Nov 19 at 16:06


















 

draft saved


draft discarded



















































 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377573%2fpython-looping-through-api-and-writing-dict-to-csv%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Tonle Sap (See)

I get strange results when I access the Sqlitedatabase with Unity C# via XAMPP

Guatemaltekische Davis-Cup-Mannschaft