Stats: 3,190,565 members, 7,941,150 topics. Date: Thursday, 05 September 2024 at 06:26 PM |
Nairaland Forum / Science/Technology / Programming / How Facebook Manages It Users Database (3672 Views)
DHTMLSQL - A Very Advanced Wrapper For Mysql Database! / Pls How Do I Get A School Software That Manages Students Data? / Laravel:call To Undefined Method Illuminate\database\eloquent\collection::save() (2) (3) (4)
How Facebook Manages It Users Database by rushman003: 8:08am On Jan 15, 2016 |
I'm still a leaner when it's comes to programming, I uses php and mysql, and a perpetual face booker *lol*. But I could help myself whenever I remembered the quantity of users that use Facebook everyday. My little knowledge about database is that the more you have users the slower your site becomes. Even I doubt Oracle could manage 100million users and think of the number of users we have on Facebook. So this challenge is up for pro programmers in the house. What database can you use to manage site like Facebook that has up to 400million users or how do you use mysql to manage it. Do you place this database on different server and users request is sent to a server that has not been saturated? 1 Like 1 Share |
Re: How Facebook Manages It Users Database by enmacmedia: 9:17am On Jan 15, 2016 |
Oracle can manage a lot more than 100 Million user. That's what most big companies use. Facebook still uses Mysql, but not exactly the Mysql we all use. They've done a lot of customization on it, the same way they have using customized php (and called it Hack). Facebook has about 1.4 Billion users not 400 Million. They have multiple servers in different parts of the world, so the load is distributed. Did I answer your question? 1 Like |
Re: How Facebook Manages It Users Database by Booyakasha(f): 9:50am On Jan 15, 2016 |
rushman003:NoSql. Google it. |
Re: How Facebook Manages It Users Database by Nobody: 11:41am On Jan 15, 2016 |
Booyakasha:. . .and that is the correct answer, you beat me to it. 1 Like |
Re: How Facebook Manages It Users Database by rushman003: 11:55am On Jan 15, 2016 |
Emmac media I understand you but you have not answered me. I think booyakhasa answered it. But it's diverted to NoSql what if I want to use MySql and expand it base.. I see you dhtml |
Re: How Facebook Manages It Users Database by Nobody: 1:38pm On Jan 15, 2016 |
^^^That one na mission impossible: Rogue Nation |
Re: How Facebook Manages It Users Database by sinequanon: 2:19pm On Jan 15, 2016 |
Don't confuse "database connection" and "user". When you log in, the server gives you a connection from a connection pool. The user associated with this connection has nothing to do with your Facebook user. It is possible that everyone uses the same user, who connects multiple times. The pool will have a connection limit. Only thousands, possibly. That is how speed (performance) is managed. If this runs out, a server busy message will propagate back to the user. A slightly different question concerns database consistency and concurrency. That is where NoSQL comes in. It allows for the database to be distributed on different domains, at the expense of temporary inconsistency. This means that users can log in independently on different domains, and your connection limit scales by the number of such domains. That is my understanding. |
Re: How Facebook Manages It Users Database by DonSegmond(m): 4:15pm On Jan 15, 2016 |
rushman003: You normalize your data, that's the first step. Avoid data duplication. For very large sites like facebook,they use some NOSQL and distributed databases, ie, Hadoop, Cassandra, etc. Google has there own custom DB, like BigTable. To use a generic database like PostgresSQL,one will shard the data, meaning divide it across multiple servers. 100million records is not really a lot of records. The problem is that most developers are pretty much one dimensional. Scaling is not a programming skill, it's a knowledge skill, so when you have such problems, you will hire a tuning expert with solid experience in system, db and scaling administration. |
Re: How Facebook Manages It Users Database by spikesC(m): 5:19am On Jan 16, 2016 |
enmacmedia: This is your answer. Mysql is able to handle billions of records and facebook use it for data that the users access. So yes, Mysql powers Facebook per say (though a heavily modified version and well distributed/load balanced) They also use other kinds of databases for different tasks and data. Booyakasha: NoSql is not the answer. It's just a different kind of data storage model. It still needs to be tuned and distributed while your data grows. rushman003: It is very possible and practical and most companies do so. You just have to tune, distribute and balance. Another best bet is also to use a cache server in front of the database(s), like memcached. sinequanon: Actually, any database can be distributed, you just have to denormalize and separate data concerns (example, using different servers for different kinds of data, which are all still used in a single page call) 4 Likes |
Re: How Facebook Manages It Users Database by Nobody: 7:15am On Jan 16, 2016 |
^^^Hmn, very true,this should be voted as the best answer. But wait, there is no voting button to click, what the heck, am out of here. |
Re: How Facebook Manages It Users Database by rushman003: 8:47am On Jan 16, 2016 |
Thanks @spikeC. Your answer explains better. That exactly where I'm going to denormalize and balance data across database which is on different server. So if a user request automatically it picks up database server that has not reach its limit and drop it there ![]() @dhtml don't start oooo. Before oga seun reason that you are planning coup de'ta and you know say that one name indefinite ban ![]() ![]() |
Re: How Facebook Manages It Users Database by Nobody: 10:01am On Jan 16, 2016 |
Wait, i ehm, apologize for my trollin behaviour. 1 Like |
Re: How Facebook Manages It Users Database by sinequanon: 12:10pm On Jan 16, 2016 |
spikesC: It is not that simple. For some systems, referential integrity needs to be maintained, and you cannot denormalize the bulk of the data. |
Re: How Facebook Manages It Users Database by spikesC(m): 12:59pm On Jan 16, 2016 |
sinequanon: I never said it's all simple. Referential integrity can be maintained in the codebase or even with stored procedures. Data normalization is a design choice |
Re: How Facebook Manages It Users Database by sinequanon: 1:08pm On Jan 16, 2016 |
spikesC: Referential integrity cannot be maintained in the code base. Stored procedures would be very clumsy and inefficient, even if you could access data on another server with your stored procedure. Data denormalization is a design choice, but it is wouldn't help solve the distribution problem of some systems. |
Re: How Facebook Manages It Users Database by Nobody: 6:34pm On Jan 16, 2016 |
^^^the dudes above me are speaking in a strange language i do not understand. . . .i need a someone to translate all those jargons to language wey lay-man/layabout-programmer/troll-programmer fit understand. |
Re: How Facebook Manages It Users Database by rushman003: 6:44pm On Jan 16, 2016 |
@Dhtml you need to see my face when I was reading their conversation. Na only cry I can no fit cry..... wrong call.... least I forget! Mod please move this thing to front page oooooo! We need more people to contribute this is very paramount to programmers as we have application that has millions of users on it. How do they expand the database!? |
Re: How Facebook Manages It Users Database by Nobody: 6:47pm On Jan 16, 2016 |
cloud database is the best for large dataset |
Re: How Facebook Manages It Users Database by spikesC(m): 1:38am On Jan 17, 2016 |
sinequanon: You realize that not all databases are RDBMS. How do you maintain references with NoSql databases or document based databases? Facebook in particular uses hive for some backend data warehousing tasks. It's sql like database which is based on hadoop. There's literally no data-based referential integrity. You have to maintain that in your code. Even RDBMS databases are decoupled just to maintain reliability and performance (everyone knows JOINS are very costly). Stored procedures are inefficient, yes, but it has always been used to maintain data integrity right from time when we didn't have foreign keys. Database developers write the codes and give the application developers the api. Super simple stuff. Ofcourse it wouldn't solve every problem but it's a start in the right way if your system permits. And it can get you a huge increase in performance (this article can help explain: https://msdn.microsoft.com/en-us/library/cc505841.aspx or search 'solving database problems with denormalization') |
Re: How Facebook Manages It Users Database by sinequanon: 2:35am On Jan 17, 2016 |
spikesC: You were the one arguing that "NoSQL is not the answer". I think you have lost track of the point. I was saying WHY NoSQL is useful. It is because it is better suited to TEMPORARY inconsistency and loss of referential integrity. Referential integrity does not require any joins. You are confusing two concepts. spikesC: You are missing the point. Stored procedures were not intended to maintain referential integrity of data distributed across boxes. That is why I was saying that NoSQL is a better solution. spikesC: I don't know why you are arguing. Some systems lend themselves easily to distributive systems. My point is that distributing other systems is intractable. I once worked on a migration project where they discovered that they data could not be decoupled. They nearly decided to can the project, which had already cost 10 million dollars. We really had to jump through hoops to migrate the data, and it was a good job that the data was only temporarily distributed for a few months, while it was being migrated. |
Re: How Facebook Manages It Users Database by Nobody: 3:08am On Jan 17, 2016 |
Geun geun, big big programmer thangs don start. |
Re: How Facebook Manages It Users Database by tpia100k: 3:11am On Jan 17, 2016 |
Re: How Facebook Manages It Users Database by Nobody: 3:23am On Jan 17, 2016 |
All these una grammers don big pass me abeg, anyway I don write one small MySQL library like that - for the small boys on the thread to look at - but all these big boys speaking big grammer, na beg i beg o. They fit come start to dey ask now if my library dey use memcache, dey cache and re-normalize referential integrity - i no sabi the answer of all those ones o. |
Re: How Facebook Manages It Users Database by seunoni34(m): 1:19pm On Jan 17, 2016 |
rushman003:Facebook stores textual data(status updates , notes etc) in mysql, and blob data (pictures, videos etc) in one of Apache Cassandra, Hadoop or apache hive (I just know this is the aspect where they use NoSQL the most) . You can check their developer page for more info. |
Re: How Facebook Manages It Users Database by sinequanon: 1:22pm On Jan 17, 2016 |
dhtml18: Hi dhtml, refer.ential integrity basically means that if one record on the database refers to the key of another record, then the other record exists on the database. The database would not let you delete the refer.enced record, for example, while another record refers to it. For example, imagine you had a table called "PAYMENT", and each PAYMENT referred to a "PAYEE". You wouldn't want to read a PAYMENT record, try to look up the PAYEE, only to find that there is no PAYEE with the given key on the PAYEE table. But this could easily happen if a program deletes a PAYEE record without checking if there are outstanding PAYMENTS for that PAYEE. It could also happen if a program creates a PAYMENT and a PAYEE record and tries to insert them into the database, but the PAYEE record fails to insert. All manner of poor program design or program bug can mean that the PAYEE is missing. By time you have discovered where the problem is in your program, and fixed the problem, your database may have a lot of missing data. i.e your database lacks "integrity". You can design your database so that it won't let your program make such mistakes. e.g if PAYEE fails to insert, it will throw out PAYMENT, too. It is very important to learn and know how to use these jargon terms. Programmers remember basic facts and material, but spend a fair amount of time looking up specifications and reference documents only as required. This becomes a key skill, and is only possible if you pay good attention to jargon, because jargon (and acronyms) are used everywhere. Without it, a five minute task of checking some basic fact and behaviour could become a project of a few days. 1 Like |
Re: How Facebook Manages It Users Database by Nobody: 3:19pm On Jan 17, 2016 |
Brings out my slate and chalk, I must learn all these jargons today today! This is really interesting. The truth is that all these things, I somehow manage to code them into my apps possibly accidentally. |
Re: How Facebook Manages It Users Database by spikesC(m): 3:36pm On Jan 17, 2016 |
sinequanon: We were probably misunderstanding each other, but i get where you're coming from now. I was giving my opinion on how mysql can be performance-tuned, while you're providing a different solution, being the use of NoSql DBs. My bad though ![]() |
Re: How Facebook Manages It Users Database by Nobody: 5:07pm On Jan 17, 2016 |
All these big big grammers don break my head finally ![]() ![]() ![]() ![]() ![]() ![]() ![]() 1 Like |
Re: How Facebook Manages It Users Database by rushman003: 5:58pm On Jan 17, 2016 |
Tight ham together dhtml! ....... learning mode activated ![]() |
Re: How Facebook Manages It Users Database by Nobody: 7:26pm On Jan 17, 2016 |
It is all over, I give up, I am going back to farming in my village. |
Re: How Facebook Manages It Users Database by Nobody: 7:43pm On Jan 17, 2016 |
dhtml18: i will dash you my 'hoe ![]() ![]() ![]() ![]() |
Re: How Facebook Manages It Users Database by Karlebolu(m): 8:08pm On Jan 17, 2016 |
100 Days Android Developer Challenge For Beginners / Why Do Seun Dont Like Php And Mysql Programmers / Asalimpo and Fincoapp - 2 excellent programmers!
(Go Up)
Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health religion celebs tv-movies music-radio literature webmasters programming techmarket Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 60 |