pcguru1:
Hi all, my favourite people on NL , I was curious about something, Someone once did something with Nairaland, they created an API around it, I am curious about the logging in, because I am experimenting with something I am not sure is possible, The goal is to see how one can scrape NL in order to modify all posts in a Scheduled manner.

naijasensei:


An API? An official API? I believe the posting stuff can either be done with bots, or some some of job scheduler working on queued jobs.

pcguru1:


Yeah that's the concept , I know there is no NL API, but I think someone wrote an API that used a scraper to fetch contents, but I want the bot to login and scrape all my contents and also make HTTP Post to modify all my content to something else. I feel one day our contents will be used against us, in this liberal world. so it will be good to be able to clear things pro-grammatically on demand.

bedfordng:
well, this can be done if you know basic python scrapping

pcguru1:


Yeah that's the concept , I know there is no NL API, but I think someone wrote an API that used a scraper to fetch contents, but I want the bot to login and scrape all my contents and also make HTTP Post to modify all my content to something else. I feel one day our contents will be used against us, in this liberal world. so it will be good to be able to clear things pro-grammatically on demand.

BlueBanana:

Sha make Se'un no catch you

bedfordng:
well, this can be done if you know basic python scrapping

pcguru1:


Handling login and session with a bot is mostly my concern now. The rest will follow

Virgo01:

Hello, I sent you a PM a day or so ago...have been following your diary for a while and want your advice on getting a new laptop. There's one you mentioned in your diary and it piqued my interest since then, can't really figure out the post now. Don't know if you'll indulge me on that.

pcguru1:


I have done scrapping in Java with JCrawler but that was to download music illegally, I do not know how to write Python.

pcguru1:

Sorry let me reply

pcguru1:
Hi all, my favourite people on NL , I was curious about something, Someone once did something with Nairaland, they created an API around it, I am curious about the logging in, because I am experimenting with something I am not sure is possible, The goal is to see how one can scrape NL in order to modify all posts in a Scheduled manner.

pcguru1:


Yeah that's the concept , I know there is no NL API, but I think someone wrote an API that used a scraper to fetch contents, but I want the bot to login and scrape all my contents and also make HTTP Post to modify all my content to something else. I feel one day our contents will be used against us, in this liberal world. so it will be good to be able to clear things pro-grammatically on demand.

stanliwise:

But you know I am quoting your post now, which means you can’t delete this post except you destroy Nairaland server especially if not backed up

MrJavaS:
I guess nairaland doesn't use stateless authentication

pcguru1:


At all, still seems to be cookies and sessions, that's where my headache stems from, luckily Elvis wrote something ill go through it. Java is what I intend to develop it in.

pcguru1:
Hi all, my favourite people on NL , I was curious about something, Someone once did something with Nairaland, they created an API around it, I am curious about the logging in, because I am experimenting with something I am not sure is possible, The goal is to see how one can scrape NL in order to modify all posts in a Scheduled manner.

pcguru1:


Yeah that's the concept , I know there is no NL API, but I think someone wrote an API that used a scraper to fetch contents, but I want the bot to login and scrape all my contents and also make HTTP Post to modify all my content to something else. I feel one day our contents will be used against us, in this liberal world. so it will be good to be able to clear things pro-grammatically on demand.

JFOD:


I have something for the first part "Getting all your contents", fell asleep so I couldn't see the result (most (if not all) of your contents should be gotten).

It's not fancy , however it gets the job done.

I can add the modifying part to it later

pcguru1:


Yeah that's the concept , I know there is no NL API, but I think someone wrote an API that used a scraper to fetch contents, but I want the bot to login and scrape all my contents and also make HTTP Post to modify all my content to something else. I feel one day our contents will be used against us, in this liberal world. so it will be good to be able to clear things pro-grammatically on demand.

JFOD:
The modifying part has been added.

Tested the Content getting, it works fine tho 2% has missing Titles/Body. I did not look into that

All your content took less than 30 minutes, tho that can be reduced.

The modifying part works fine

pcguru1:


What did you write it in ? Java ?

JFOD:


Python (Selenium and Beautiful Soup).

Qrash:
Good evening sir long time. How is your health now