pandas Add html tables values where data is int
0
I am getting html table based on day so if I search for 20 days it brings me 20 table and I want to add all 20 tables in 1 table so I can verify data within time series.
I have tried merge and add functions of pandas but it just add as string.
Table one
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
table two
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
code but it add as string.
tab_data = [[item.text for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
df = pd.DataFrame(tab_data)
df2 = pd.DataFrame(tab_data)
df3 = df.add(df2,fill_value=0)
df
python python-3.x pandas selenium selenium-webdriver
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
add a comment |
0
I am getting html table based on day so if I search for 20 days it brings me 20 table and I want to add all 20 tables in 1 table so I can verify data within time series.
I have tried merge and add functions of pandas but it just add as string.
Table one
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
table two
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
code but it add as string.
tab_data = [[item.text for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
df = pd.DataFrame(tab_data)
df2 = pd.DataFrame(tab_data)
df3 = df.add(df2,fill_value=0)
df
python python-3.x pandas selenium selenium-webdriver
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
2
Instead of using a list comprehension, use a for loop and try to convert values in the table to numbers (float or int) in the loop. Make use oftry/exceptto handle actual strings.
– wwii
Nov 22 '18 at 22:11
Can you give me little hint even with pseudo code ?
– Ashfaque Marfani
Nov 22 '18 at 22:15
In row I have string and number mix how can i solve it ?
– Ashfaque Marfani
Nov 22 '18 at 22:16
add a comment |
0
0
0
1
I am getting html table based on day so if I search for 20 days it brings me 20 table and I want to add all 20 tables in 1 table so I can verify data within time series.
I have tried merge and add functions of pandas but it just add as string.
Table one
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
table two
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
code but it add as string.
tab_data = [[item.text for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
df = pd.DataFrame(tab_data)
df2 = pd.DataFrame(tab_data)
df3 = df.add(df2,fill_value=0)
df
python python-3.x pandas selenium selenium-webdriver
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
I am getting html table based on day so if I search for 20 days it brings me 20 table and I want to add all 20 tables in 1 table so I can verify data within time series.
I have tried merge and add functions of pandas but it just add as string.
Table one
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
table two
[['xa0', 'All Issues', 'Investment Grade', 'High Yield', 'Convertible'],
['Total Issues Traded', '8039', '5456', '2386', '197'],
['Advances', '3834', '2671', '1075', '88'],
['Declines', '3668', '2580', '994', '94'],
['Unchanged', '163', '54', '99', '10'],
['52 Week High', '305', '100', '193', '12'],
['52 Week Low', '152', '83', '63', '6'],
['Dollar Volume*', '27568', '17000', '9299', '1269']]
code but it add as string.
tab_data = [[item.text for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
df = pd.DataFrame(tab_data)
df2 = pd.DataFrame(tab_data)
df3 = df.add(df2,fill_value=0)
df
python python-3.x pandas selenium selenium-webdriver
python python-3.x pandas selenium selenium-webdriver
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
edited Nov 22 '18 at 22:17
Ashfaque Marfani
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
asked Nov 22 '18 at 22:04
Ashfaque MarfaniAshfaque Marfani
6912
6912
2
Instead of using a list comprehension, use a for loop and try to convert values in the table to numbers (float or int) in the loop. Make use oftry/exceptto handle actual strings.
– wwii
Nov 22 '18 at 22:11
Can you give me little hint even with pseudo code ?
– Ashfaque Marfani
Nov 22 '18 at 22:15
In row I have string and number mix how can i solve it ?
– Ashfaque Marfani
Nov 22 '18 at 22:16
add a comment |
2
Instead of using a list comprehension, use a for loop and try to convert values in the table to numbers (float or int) in the loop. Make use oftry/exceptto handle actual strings.
– wwii
Nov 22 '18 at 22:11
Can you give me little hint even with pseudo code ?
– Ashfaque Marfani
Nov 22 '18 at 22:15
In row I have string and number mix how can i solve it ?
– Ashfaque Marfani
Nov 22 '18 at 22:16
2
2
Instead of using a list comprehension, use a for loop and try to convert values in the table to numbers (float or int) in the loop. Make use of
try/except to handle actual strings.– wwii
Nov 22 '18 at 22:11
Instead of using a list comprehension, use a for loop and try to convert values in the table to numbers (float or int) in the loop. Make use of
try/except to handle actual strings.– wwii
Nov 22 '18 at 22:11
Can you give me little hint even with pseudo code ?
– Ashfaque Marfani
Nov 22 '18 at 22:15
Can you give me little hint even with pseudo code ?
– Ashfaque Marfani
Nov 22 '18 at 22:15
In row I have string and number mix how can i solve it ?
– Ashfaque Marfani
Nov 22 '18 at 22:16
In row I have string and number mix how can i solve it ?
– Ashfaque Marfani
Nov 22 '18 at 22:16
add a comment |
2 Answers
2
active
oldest
votes
1
If you want to convert the numeric cells into integers, you would need to do that explicitly, as follows:
tab_data = [[int(item.text) if item.text.isdigit() else item.text
for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
Hope it helps.
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
add a comment |
1
The way you are converting the data frame treats all values as text.
There are two options here.
- Explicitly convert the strings to the data type you want using astype
- Use read_html to create data frames from html tables, which also tries to do the data type conversion.
answered Nov 22 '18 at 22:42
hmadhmad
863
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438454%2fpandas-add-html-tables-values-where-data-is-int%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
If you want to convert the numeric cells into integers, you would need to do that explicitly, as follows:
tab_data = [[int(item.text) if item.text.isdigit() else item.text
for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
Hope it helps.
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
add a comment |
1
If you want to convert the numeric cells into integers, you would need to do that explicitly, as follows:
tab_data = [[int(item.text) if item.text.isdigit() else item.text
for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
Hope it helps.
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
add a comment |
1
1
1
If you want to convert the numeric cells into integers, you would need to do that explicitly, as follows:
tab_data = [[int(item.text) if item.text.isdigit() else item.text
for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
Hope it helps.
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
If you want to convert the numeric cells into integers, you would need to do that explicitly, as follows:
tab_data = [[int(item.text) if item.text.isdigit() else item.text
for item in row_data.select("th,td")]
for row_data in tables.select("tr")]
Hope it helps.
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
answered Nov 22 '18 at 23:07
TeeKeaTeeKea
3,20841730
3,20841730
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
add a comment |
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
Thankyou so much, I did worked for int cells. :)
– Ashfaque Marfani
Nov 23 '18 at 11:06
add a comment |
1
The way you are converting the data frame treats all values as text.
There are two options here.
- Explicitly convert the strings to the data type you want using astype
- Use read_html to create data frames from html tables, which also tries to do the data type conversion.
answered Nov 22 '18 at 22:42
hmadhmad
863
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
add a comment |
1
The way you are converting the data frame treats all values as text.
There are two options here.
- Explicitly convert the strings to the data type you want using astype
- Use read_html to create data frames from html tables, which also tries to do the data type conversion.
answered Nov 22 '18 at 22:42
hmadhmad
863
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
add a comment |
1
1
1
The way you are converting the data frame treats all values as text.
There are two options here.
- Explicitly convert the strings to the data type you want using astype
- Use read_html to create data frames from html tables, which also tries to do the data type conversion.
answered Nov 22 '18 at 22:42
hmadhmad
863
The way you are converting the data frame treats all values as text.
There are two options here.
- Explicitly convert the strings to the data type you want using astype
- Use read_html to create data frames from html tables, which also tries to do the data type conversion.
answered Nov 22 '18 at 22:42
hmadhmad
863
answered Nov 22 '18 at 22:42
hmadhmad
863
answered Nov 22 '18 at 22:42
hmadhmad
863
answered Nov 22 '18 at 22:42
hmadhmad
863
863
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
add a comment |
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
Thankyou so much for your answer but read_html is not working it is same as above.
– Ashfaque Marfani
Nov 23 '18 at 11:05
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438454%2fpandas-add-html-tables-values-where-data-is-int%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Instead of using a list comprehension, use a for loop and try to convert values in the table to numbers (float or int) in the loop. Make use of
try/exceptto handle actual strings.– wwii
Nov 22 '18 at 22:11
Can you give me little hint even with pseudo code ?
– Ashfaque Marfani
Nov 22 '18 at 22:15
In row I have string and number mix how can i solve it ?
– Ashfaque Marfani
Nov 22 '18 at 22:16