Welcome, Guest: Register On Nairaland / LOGIN! / Trending / Recent / NewStats: 3,201,184 members, 7,977,468 topics. Date: Thursday, 17 October 2024 at 08:48 AM |
Nairaland Forum / Science/Technology / Programming / Nairaland's February Front Page Data Analysis (1727 Views)
Analysis Of The Nairaland's Front Page Topics, Using Python And Tableau / Please Rate My Nairaland.com Front Page Mock Design In HTML/CSS/JS / Programming Not Enough Anymore: Data Analysis, Ml And A.i Is The Future. (2) (3) (4)
Nairaland's February Front Page Data Analysis by Felixitie(m): 9:57pm On Mar 03, 2022 |
Yeah,I was trying to check my proficiency in data analysis, I decided to use the first 10 pages on Nairaland front page topics. Web scraping( selemium, Bs4 , HTMLsession, Scrapy) , Python Pandas, PowerBI , Excel, SQL I Web scraped the data using python Beautifulsoup (Bs4). This was done by getting all the front page topic links, lopped through each one of them to get ; the title of the topic, poster of the topic, time and date it was posted. The output was then saved in csv file format. I could have cleaned the raw data using python Pandas, but I prefer to used powerbi since I planned using it to create a simple dashboard for the analysis. Therefore PowerBI was used for visualization. More of descriptive analysis for the month of February ; Who posted most topic? Time of the with most topic? How does the Daily Topic trends look? 6 Likes 8 Shares
|
Re: Nairaland's February Front Page Data Analysis by Enskynelson(m): 10:35pm On Mar 03, 2022 |
Nice one. Would like to follow your work. Please check your DM. |
Re: Nairaland's February Front Page Data Analysis by Samboizz: 11:00pm On Mar 03, 2022 |
Nice one bro. I would like to get in contact with you |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 11:13pm On Mar 03, 2022 |
To further the analysis, I want to see the sections that produces top most topics on the front page for the current month of March This should give us the section that interest members most or the ones the NL management wants people to see. Based on the last 10 front page topics ( March 4th, today to the last 10 pages. 2 Likes 3 Shares
|
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 11:20pm On Mar 03, 2022 |
Enskynelson: OKAY SeniorMan |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 11:31pm On Mar 03, 2022 |
Samboizz: Responded to your pm! |
Re: Nairaland's February Front Page Data Analysis by airsaylongcome: 2:45am On Mar 04, 2022 |
Felixitie: I swear I thought he would have included that in his analysis |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 9:16am On Mar 04, 2022 |
At least 60% of those topics were posted between Monday to Thursday of the week. You still have some 40% of the topics left to grab over the weekend if you were busy going through the mid-week topics lol
|
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 2:26pm On Mar 04, 2022 |
Felixitie: It is obvious that the political section got highest topic moved to the front page.. Even the programming section is missing, is it that people aren't interested in programming or IT related field? 1 Like |
Re: Nairaland's February Front Page Data Analysis by Prinss: 7:03pm On Mar 04, 2022 |
Nice initiative 1 Like |
Re: Nairaland's February Front Page Data Analysis by Aybee92(m): 7:36pm On Mar 04, 2022 |
Nice work. Keep it up 1 Like |
Re: Nairaland's February Front Page Data Analysis by LikeAking: 9:28pm On Mar 04, 2022 |
Oga u too much. NL should just employ you. Nice! You are finishing werk. 1 Like |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 12:33am On Mar 05, 2022 |
LikeAking: LOL. We are just trying to cruise with the little skills we learn. Thanks boss man. 1 Like 1 Share |
Re: Nairaland's February Front Page Data Analysis by Chuukwudi(m): 7:58am On Mar 05, 2022 |
How did you get the data? Does nairaland have an API or you manually read the lines and reg expressions to get dates and all? |
Re: Nairaland's February Front Page Data Analysis by extendo: 10:32am On Mar 05, 2022 |
Chuukwudi:NL does not have an API. You have to scrape and parse whatever data you need using libraries like scrapy/beautifulsoup/regex. 5 Likes |
Re: Nairaland's February Front Page Data Analysis by Kenny242(m): 12:04pm On Mar 05, 2022 |
Wow good work op On my way too Lalasticlala 1 Like |
Re: Nairaland's February Front Page Data Analysis by Chuukwudi(m): 12:42pm On Mar 05, 2022 |
extendo: Nice. If the op Felixitie would let me have the csv file, I might be able to predict the most likely section subsequent front page topics will emerge from. This can Infact be used to score the admins of all the sections and classify them in a way that reflects what section Nairalanders are most interested in, the job of the mods, etc. I am sure I will find high correlation between snake threads and lalasticlala. I wonder if I've got time for that. But it would be a nice idea for anyone trying to learn some Machine Learning. 2 Likes 1 Share |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 3:12pm On Mar 05, 2022 |
Chuukwudi: Web scraping bro, with Request & Beautifulsoup.. |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 3:14pm On Mar 05, 2022 |
Chuukwudi: This idea is cool bro, if you are still willing to have the csv file and work on it, I can forward it to you. Machine Learning no be beans ooo.. |
Re: Nairaland's February Front Page Data Analysis by airsaylongcome: 8:29pm On Mar 05, 2022 |
Felixitie: What percentage of FP topics were from Programming sub? Programming threads have started hitting fp now a bit more than before. Cam you extend your data harvesting to see percentage increase of Programming threads? |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 9:47pm On Mar 05, 2022 |
airsaylongcome: Yes sure I can,apparently it is a nice insight to look at. BTW,do u mean percentage difference in month on month? And from which month specifically do u want me to look at ? Jan - February or December - January etc. I hope you understand! |
Re: Nairaland's February Front Page Data Analysis by airsaylongcome: 10:53pm On Mar 05, 2022 |
Felixitie: December 2021 to January 2022 would be interesting. Just to see the percentage difference in Programming Section thread that had hit FP. I'm a data lover and with the data you have to many thoughts have come to my head. Which section noticed a decline in threads hitting FP? |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 11:10pm On Mar 05, 2022 |
airsaylongcome: Definitely I should be able to come up with that information. I will extend my scraping to December. Obviously we have a lot of insights to derive from the dataset.. but for what purpose actually? NL no send me message.. Lol 1 Like |
Re: Nairaland's February Front Page Data Analysis by Gentleman001: 7:56pm On Mar 06, 2022 |
Kudos brother.. Checking the url of each topic, I noticed they fellow different pattern( unlike some web where you can easily loop through multiple pages with range of page numbers to perform the scraping )... My question is, do you manually copied all the url link you scraped or there is a way you go about it? |
Re: Nairaland's February Front Page Data Analysis by mustyx001: 8:14pm On Mar 06, 2022 |
Wow I also used web scraping to redesign Nairaland into a new app. I used PHP / Laravel and React js This topic of yours will look like this ( check the pictures) Link at : https://nairaland-v2.netlify.app/topic/7013550/nairalands-february-front-page-data 1 Like
|
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 8:55pm On Mar 06, 2022 |
Gentleman001: This is how it works, with beautifulsoup. You get or scrap the link for all the topics you need in form of a list and have it assigned a variable, then you loop through each of them. Those links have the same html setup or tags.. |
Re: Nairaland's February Front Page Data Analysis by mustyx001: 9:18pm On Mar 06, 2022 |
Felixitie: Nice one. Just curious, can request and beautifulsoup bypass Amazon's anti-scraping mechanism. Scraping Nairaland with php also went smoothly since they didn't prevent scraping. |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 10:00pm On Mar 06, 2022 |
mustyx001: Selenium works, then pass it to beautifulsoup |
Re: Nairaland's February Front Page Data Analysis by wellrounded: 2:37am On Mar 08, 2022 |
[i][/i]Thank you Felixitie: |
Re: Nairaland's February Front Page Data Analysis by Anon4: 8:54pm On Mar 08, 2022 |
mustyx001:it takes time to load lol.. |
Re: Nairaland's February Front Page Data Analysis by Anon4: 8:55pm On Mar 08, 2022 |
Felixitie:do you run it on cloud or locally |
Re: Nairaland's February Front Page Data Analysis by Felixitie(m): 9:28pm On Mar 08, 2022 |
Anon4: On my local machine! |
I Need Someone Who Will Teach Me Java And Sql Server 2008 / Creating A Startup Section / Sudoku And Linear Algebra
Viewing this topic: 1 guest(s)
(Go Up)
Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health religion celebs tv-movies music-radio literature webmasters programming techmarket Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 51 |