Stats: 3,190,515 members, 7,940,951 topics. Date: Thursday, 05 September 2024 at 02:20 PM |
Nairaland Forum / Science/Technology / Programming / Solving Real Problems (6317 Views)
Solving The Nigerian Budget Padding Issue With PHP (2) (3) (4)
Solving Real Problems by seunthomas: 8:30pm On Feb 25, 2016 |
Hi guys am creating this thread to teach how to solve basic programming challenges. Please i would rather avoid unruely manner in this thread. Its specifically for learning only. |
Re: Solving Real Problems by seunthomas: 8:34pm On Feb 25, 2016 |
First question: Nairaland is a top forum/social media site in Nigeria. From time to time nairaland experiences downtime or database/server related issues. Profer solutions for how we can make nairaland more stable. Lets make the assumptions that the programmers at nairaland are near perfect and there is no bug in their code. The only challenge would be limited resources. Nairaland server is run on a single quadcore server with 16GB ram. They use hard drives instead of ssd. At peak periods nairaland experiences about 10-100k requests: a. How can we ensure nairaland serves requests faster b. what solution for faster database application response c. any other cool idea to scale this site |
Re: Solving Real Problems by omohayek: 10:02pm On Feb 25, 2016 |
When you say that at peak periods nairaland experiences about 10-100k requests, what time frame are you talking about? Per second, minute, hour or day? Without that information it's hard to say anything meaningful about scalability requirements - there are 86,400 seconds in a day, after all. In addition, a lot more specificity about the technologies used would be helpful. For instance, a Java-based site should be able to handle 3-10 times the load of one written in pure Python or Ruby, and that's even without concurrency considerations. Finally, I don't see how any site owner that's serious about availability would ever choose to host their site on just a single server, especially one built with the components of a typical gaming rig. All those single points of failure within the machine are just asking for disaster - and that's not even considering the possibility of a failing router or a fire at the hosting center. For reliability reasons, it would even be preferable to spec up several lower-end machines distributed across several regions than to use a single such machine, and renting from Amazon or Heroku isn't that expensive for hosting boxes serving mostly static content. |
Re: Solving Real Problems by seunthomas: 10:34pm On Feb 25, 2016 |
omohayek:Thanks for your contribution. The peak requests is measured per second. As for the technology used, it could be any platform(php,python,java,ruby,etc). Scalability is what we are trying to deal with here and i am sure you are aware that would depend on a lot of factors. But so as not to go to far off, lets stick to the itemized points. a. How can we ensure nairaland serves requests faster(from a web server or application server point of view) b. what solution for faster database application response c. any other cool idea to scale this site |
Re: Solving Real Problems by omohayek: 10:52pm On Feb 25, 2016 |
seunthomas: You can't really divorce scalability from your choice of platform: for example, given the existence of global interpret locks in both Python (CPython variant) and Ruby, there's only so much you can do concurrency-wise by comparison with Java or C#/.Net, and that's on top of the fact that JVM-optimized code can an order of magnitude faster than interpreted code. Java is no. 1 on the back end for very good reasons. Now, for some concrete suggestions on scaling the site out: 1. Deploy a distributed in-memory caching system for queries, e.g. Memcached or Redis 2. For storage, use a distributed data store like Amazon DynamoDB or Riak, instead of a traditional RDBMS. 3. Offload serving of images and other such content using Nginx/Varnish - or simply pay Cloudflare to do the same thing for you. This website doesn't really have any CPU-intensive content, so I'd also rent lots of smaller Amazon/Azure server instances with more memory, instead of 1 bigger one like your original configuration. Now, I'd like to see your own contribution, instead of just setting others challenges. How exactly would you propose to deploy your platform in a scalable way, across potentially hundreds of servers? 2 Likes |
Re: Solving Real Problems by seunthomas: 11:07pm On Feb 25, 2016 |
omohayek:Well i like your contribution so far but me i prefer to save my employers/client on money spent on hardware or vps/cloud servers. I know we can achieve more with platform but i dont want us to disturb the already existing application. Our solution should just require very little changes to the existing application which is always a preferred way. a. Instead of using a threaded server use an asynchronous server, this use more cpu cycles but can handle more load than threaded servers like apache. Like you rightly suggested nginx or haproxy with nginx. b. You are thinking like me on the database level. But assuming Seun does not want to change the application using a key store would be a challenge cos am sure he is using an RDBMS. c. My best solution would be to use docker on the 32GB server and setup 3-5 haproxy nodes,5> mysql nodes as a cluster and will setup 3-5 nginx instances to serve only static content. Will elaborate more on this solution. |
Re: Solving Real Problems by seunthomas: 10:26am On Feb 26, 2016 |
New Problem How can a site like NL ensure that when we post stuff its realtime or near realtime. Proposal develop a chat system for NL or websocket based realtime engine for NL. The realtime engine will support like,comments,post and even chat. Any ideas? |
Re: Solving Real Problems by webdevgod: 10:34am On Feb 26, 2016 |
seunthomas: Redevelop the entire site using AngularJS to make the site dynamic |
Re: Solving Real Problems by seunthomas: 10:39am On Feb 26, 2016 |
webdevgod:Nice try but the site is already dynamic. So explain more. The solution should be a seperate system for real time data. |
Re: Solving Real Problems by omohayek: 10:41am On Feb 26, 2016 |
webdevgod: I don't see how this is a solution to the problem at hand. Isn't creating the right back-end architecture the more crucial task here? Last I heard, AngularJS was a front-end framework. |
Re: Solving Real Problems by Olumyco(m): 11:13am On Feb 26, 2016 |
omohayek: Yes Angular is a front end framework.... better put Angular is a front end MVVM Architecture framework... It uses client - server technology. .. It makes sense... |
Re: Solving Real Problems by Olumyco(m): 11:35am On Feb 26, 2016 |
If we are proposing to build a Chat system or web socket real time engine for nairaland, I think we will av to pay much attention to scalability and concurrency. Some of the tins we will be doing will b how to achieve Asynchronous event loop I/O stuff for the d engine.... Then how to communicate wit d server from d background (client) JavaScript (nodeJs) can help out in dis case... Also we can use Erlang... As for JavaScript (nodeJs) we can use socket module or websocket module of nodeJs for it and spice it up with backbone (it is a light weight MVVM Architecture framework) for the "View" of our gui. |
Re: Solving Real Problems by Nobody: 2:09pm On Feb 26, 2016 |
Olumyco:Node.JS has this http://socket.io that takes care of the socket server thing. And i have also created a socket server that is the equivalent of socket.io using PHP and Basic javaScript. I have used this solution to develop. Here is a simple chat implementing NODE Socket.IO - dhtmlchat.heroku.com I once made a group chat somewhere that allows nairalanders to login with their valid nairaland account details and chat with each other. Seun blocked the website link from ever showing on nairaland. I dont know if the script still works, took it offline sometime back. |
Re: Solving Real Problems by seunthomas: 2:48pm On Feb 26, 2016 |
donjayzi:It wont scale. Trust me. Nodejs is not something that can easily do c1000k, maybe c10k or c100k(sparingly). If you are doing too much concurrency with nodejs, you would need to buy more hardware. |
Re: Solving Real Problems by Olumyco(m): 5:26pm On Feb 26, 2016 |
seunthomas: It will scale... Infact that is where it's strength is..... NodeJs has dat capacity to handle many connections.... LinkedIn, Yahoo are already using it.... |
Re: Solving Real Problems by seunthomas: 7:24pm On Feb 26, 2016 |
Olumyco:They told u they use it in production right ![]() |
Re: Solving Real Problems by Nobody: 11:43am On Feb 27, 2016 |
seunthomas: More and more javascripting, will need ajax for that. Coincidentally I am implementing this currently in my project and I'm using PHP, JavaScript and Ajax for this and I'm doing extra hard-coding to make it very scalable |
Re: Solving Real Problems by Nobody: 11:53am On Feb 27, 2016 |
Something more like this
|
Re: Solving Real Problems by Nobody: 12:04pm On Feb 27, 2016 |
Then planning a database_schema is important, if the database is ready then we can use PHP add the styling and the script.js will look like this:
Other functionalities like submitting a new chat entry, adding a chat entry to the page, dynamically appending the mark-up and adding custom error messages. This is example is not scalable, but can be made scalable. It is suitable for a customer care system tho. |
Re: Solving Real Problems by seunthomas: 10:44am On Feb 28, 2016 |
New problem on the way.... |
Re: Solving Real Problems by DonSegmond(m): 9:30pm On Feb 29, 2016 |
seunthomas: Scaling a site shouldn't be after the fact, it really should happen during development. But been that this site is just a simple web forum, which is read heavy, the scaling solution is pretty standard. Cache like hell, consider a ram disk and load the entire DB in memory for all text content, all non text content should be hosted on an outside CDN. Having Nairaland handle a million users daily should be pretty trivial. Been that you are reading from HD instead of SSD, limit the amount of reads, since HD spinning is expensive, this comes down to sysadmin duties, checking DB/system configurations. I honestly don't believe any scaling solution right now should involve coding, it's all about having a knowledge scaling/sysadmins/dba ops experience. |
Re: Solving Real Problems by seunthomas: 10:18pm On Feb 29, 2016 |
DonSegmond:Though your solution is good and effective, newer ways of application development mean you can actually handle more load than previously. Take a look at vertx and see what they can achieve with limited resources. |
Re: Solving Real Problems by seunthomas: 7:30am On Mar 05, 2016 |
I got an interesting challenge for you guys on the IOS security. Most of you who have been following the FBI vs Apple case would probably have an understanding of what this is about. The FBI has been trying to get into the iphone 5c of a couple responsible for an act of terror somewhere in the US( San Bernardino,CA). Now the FBI guru's claim they cant get into the device(i will leave the details) and hence they need apple to create a backdoor. Apple refuses. Kindly air your views. Is the FBI correct, is Apple correct. |
Re: Solving Real Problems by FincoApps(m): 7:42am On Mar 05, 2016 |
seunthomas: I don't think it is right to force Apple to create a backdoor for their device. Apple actually helped with getting some data from iCloud, however the FBI kinda messed up and reset the Phone's iCloud account. And now it seems the only way around it is for Apple to create a Backdoor. If I were in charge of Apple, I would but I still don't think the FBI should force them. |
Re: Solving Real Problems by seunthomas: 7:46am On Mar 05, 2016 |
FincoApps:But lets be real here do you think the FBI,NSA,CIA actually need help to decrypt the data? Even if they were using a 1024 bit cipher or do you think apple's OS is truly as secure as they make it out to be? I think the FBI wants something more. |
Re: Solving Real Problems by FincoApps(m): 8:15am On Mar 05, 2016 |
seunthomas: On one hand, you could be correct... On another hand I think they do need Apple's help. And it's not just decrypting data they need. They need to unlock the phone without loosing data. Normally, they should be able to unlock it, but that would mean loosing the data |
Re: Solving Real Problems by seunthomas: 8:20am On Mar 05, 2016 |
FincoApps:Ok, let me tell you my honest opinion. I think the FBI just needs legality. The point is they probably have all they need on that device. They just cant present it in court because it cant be admissible(due to the means they got it). They need legal powers to say yeah we got this through the right means and are presenting it in court else any judge would throw it out. Also they want it to be wider more constant arrangement so they dont need to go through the rigours the next time they want to prosecute a case. |
Re: Solving Real Problems by omohayek: 9:41am On Mar 05, 2016 |
seunthomas: A nice idea, but I think the real answer is even more cynical. The FBI doesn't really need the data on this phone, which belongs to a man who is now dead; the true purpose of this lawsuit is to set a precedent for the future, so that the FBI and the other three-letter agencies can force tech firms to backdoor all of their devices. That everyone in Silicon Valley understands this underlying ambition is why staunch Apple rivals like Google and Microsoft have signed amicus curiae briefs in favor of the company. If the FBI wins this case, no one's data will be safe anywhere in the world. The French are already trying to pass a law to force phone makers to decrypt devices or have their executives thrown in jail, and both the Chinese and the Russians would love to be able to pass similar laws without being accused of censorship. |
Re: Solving Real Problems by FincoApps(m): 9:48am On Mar 05, 2016 |
seunthomas: Truly it's possible. Since I don't work in either place, I'm not sure but it truly appears FBI should have the tools to do the job. For me, I really don't believe in hiding data from government Agencies at all so Apple should calm down and do what the FBI wants |
Re: Solving Real Problems by FincoApps(m): 9:49am On Mar 05, 2016 |
omohayek: Very reasonable. I hope the FBI wins the case |
Re: Solving Real Problems by danvery2k6(m): 12:53pm On Mar 05, 2016 |
1 Like |
Re: Solving Real Problems by Nobody: 3:11pm On Mar 05, 2016 |
FincoApps: Very unreasonable, Apple should win. 2 Likes |
Java Server Pages (jsp) Or Java Server Face(jsf Which Path To Trend? ? ? ? / [Help Request ] I Want To Learn Programming, what are the requirements / Any Software Product From "nairaland Programmers"?
(Go Up)
Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health religion celebs tv-movies music-radio literature webmasters programming techmarket Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 97 |