I'm still a leaner when it's comes to programming, I uses php and mysql, and a perpetual face booker *lol*. But I could help myself whenever I remembered the quantity of users that use Facebook everyday. My little knowledge about database is that the more you have users the slower your site becomes. Even I doubt Oracle could manage 100million users and think of the number of users we have on Facebook. So this challenge is up for pro programmers in the house. What database can you use to manage site like Facebook that has up to 400million users or how do you use mysql to manage it. Do you place this database on different server and users request is sent to a server that has not been saturated?

Oracle can manage a lot more than 100 Million user. That's what most big companies use. Facebook still uses Mysql, but not exactly the Mysql we all use. They've done a lot of customization on it, the same way they have using customized php (and called it Hack).

Facebook has about 1.4 Billion users not 400 Million. They have multiple servers in different parts of the world, so the load is distributed.

Did I answer your question?

rushman003:
I'm still a leaner when it's comes to programming, I uses php and mysql, and a perpetual face booker *lol*. But I could help myself whenever I remembered the quantity of users that use Facebook everyday. My little knowledge about database is that the more you have users the slower your site becomes. Even I doubt Oracle could manage 100million users and think of the number of users we have on Facebook. So this challenge is up for pro programmers in the house. What database can you use to manage site like Facebook that has up to 400million users or how do you use mysql to manage it. Do you place this database on different server and users request is sent to a server that has not been saturated?

NoSql. Google it.

Booyakasha:

NoSql. Google it.

. . .and that is the correct answer, you beat me to it.

Emmac media I understand you but you have not answered me. I think booyakhasa answered it. But it's diverted to NoSql what if I want to use MySql and expand it base.. I see you dhtml

^^^That one na mission impossible: Rogue Nation

Don't confuse "database connection" and "user".

When you log in, the server gives you a connection from a connection pool. The user associated with this connection has nothing to do with your Facebook user. It is possible that everyone uses the same user, who connects multiple times.

The pool will have a connection limit. Only thousands, possibly. That is how speed (performance) is managed. If this runs out, a server busy message will propagate back to the user.

A slightly different question concerns database consistency and concurrency. That is where NoSQL comes in. It allows for the database to be distributed on different domains, at the expense of temporary inconsistency. This means that users can log in independently on different domains, and your connection limit scales by the number of such domains.

That is my understanding.

rushman003:
I'm still a leaner when it's comes to programming, I uses php and mysql, and a perpetual face booker *lol*. But I could help myself whenever I remembered the quantity of users that use Facebook everyday. My little knowledge about database is that the more you have users the slower your site becomes. Even I doubt Oracle could manage 100million users and think of the number of users we have on Facebook. So this challenge is up for pro programmers in the house. What database can you use to manage site like Facebook that has up to 400million users or how do you use mysql to manage it. Do you place this database on different server and users request is sent to a server that has not been saturated?

You normalize your data, that's the first step. Avoid data duplication. For very large sites like facebook,they use some NOSQL and distributed databases, ie, Hadoop, Cassandra, etc. Google has there own custom DB, like BigTable. To use a generic database like PostgresSQL,one will shard the data, meaning divide it across multiple servers. 100million records is not really a lot of records. The problem is that most developers are pretty much one dimensional. Scaling is not a programming skill, it's a knowledge skill, so when you have such problems, you will hire a tuning expert with solid experience in system, db and scaling administration.

enmacmedia:
Oracle can manage a lot more than 100 Million user. That's what most big companies use. Facebook still uses Mysql, but not exactly the Mysql we all use. They've done a lot of customization on it, the same way they have using customized php (and called it Hack).
Facebook has about 1.4 Billion users not 400 Million. They have multiple servers in different parts of the world, so the load is distributed.
Did I answer your question?

This is your answer. Mysql is able to handle billions of records and facebook use it for data that the users access. So yes, Mysql powers Facebook per say (though a heavily modified version and well distributed/load balanced)
They also use other kinds of databases for different tasks and data.

Booyakasha:

NoSql. Google it.

NoSql is not the answer. It's just a different kind of data storage model. It still needs to be tuned and distributed while your data grows.

rushman003:
...what if I want to use MySql and expand it base...

It is very possible and practical and most companies do so. You just have to tune, distribute and balance. Another best bet is also to use a cache server in front of the database(s), like memcached.

sinequanon:
...
A slightly different question concerns database consistency and concurrency. That is where NoSQL comes in. It allows for the database to be distributed on different domains, at the expense of temporary inconsistency. This means that users can log in independently on different domains, and your connection limit scales by the number of such domains.
That is my understanding.

Actually, any database can be distributed, you just have to denormalize and separate data concerns (example, using different servers for different kinds of data, which are all still used in a single page call)

^^^Hmn, very true,this should be voted as the best answer. But wait, there is no voting button to click, what the heck, am out of here.

Thanks @spikeC. Your answer explains better. That exactly where I'm going to denormalize and balance data across database which is on different server. So if a user request automatically it picks up database server that has not reach its limit and drop it there ......
@dhtml don't start oooo. Before oga seun reason that you are planning coup de'ta and you know say that one name indefinite ban ... Leave oga seun make him decide on wetin to put for his invention.

Wait, i ehm, apologize for my trollin behaviour.

spikesC:
Actually, any database can be distributed, you just have to denormalize and separate data concerns (example, using different servers for different kinds of data, which are all still used in a single page call)

It is not that simple. For some systems, referential integrity needs to be maintained, and you cannot denormalize the bulk of the data.

sinequanon:


It is not that simple. For some systems, referential integrity needs to be maintained, and you cannot denormalize the bulk of the data.

I never said it's all simple. Referential integrity can be maintained in the codebase or even with stored procedures. Data normalization is a design choice

spikesC:


I never said it's all simple. Referential integrity can be maintained in the codebase or even with stored procedures. Data normalization is a design choice

Referential integrity cannot be maintained in the code base.

Stored procedures would be very clumsy and inefficient, even if you could access data on another server with your stored procedure.

Data denormalization is a design choice, but it is wouldn't help solve the distribution problem of some systems.

^^^the dudes above me are speaking in a strange language i do not understand. . . .i need a someone to translate all those jargons to language wey lay-man/layabout-programmer/troll-programmer fit understand.

@Dhtml you need to see my face when I was reading their conversation. Na only cry I can no fit cry..... wrong call.... least I forget! Mod please move this thing to front page oooooo! We need more people to contribute this is very paramount to programmers as we have application that has millions of users on it. How do they expand the database!?

cloud database is the best for large dataset

sinequanon:


Referential integrity cannot be maintained in the code base.

Stored procedures would be very clumsy and inefficient, even if you could access data on another server with your stored procedure.

Data denormalization is a design choice, but it is wouldn't help solve the distribution problem of some systems.

You realize that not all databases are RDBMS. How do you maintain references with NoSql databases or document based databases?
Facebook in particular uses hive for some backend data warehousing tasks. It's sql like database which is based on hadoop. There's literally no data-based referential integrity. You have to maintain that in your code. Even RDBMS databases are decoupled just to maintain reliability and performance (everyone knows JOINS are very costly).

Stored procedures are inefficient, yes, but it has always been used to maintain data integrity right from time when we didn't have foreign keys. Database developers write the codes and give the application developers the api. Super simple stuff.

Ofcourse it wouldn't solve every problem but it's a start in the right way if your system permits. And it can get you a huge increase in performance (this article can help explain: https://msdn.microsoft.com/en-us/library/cc505841.aspx or search 'solving database problems with denormalization')

spikesC:


You realize that not all databases are RDBMS. How do you maintain references with NoSql databases or document based databases?
Facebook in particular uses hive for some backend data warehousing tasks. It's sql like database which is based on hadoop. There's literally no data-based referential integrity. You have to maintain that in your code. Even RDBMS databases are decoupled just to maintain reliability and performance (everyone knows JOINS are very costly).

You were the one arguing that "NoSQL is not the answer". I think you have lost track of the point.

I was saying WHY NoSQL is useful. It is because it is better suited to TEMPORARY inconsistency and loss of referential integrity.

Referential integrity does not require any joins. You are confusing two concepts.

spikesC:
Stored procedures are inefficient, yes, but it has always been used to maintain data integrity right from time when we didn't have foreign keys. Database developers write the codes and give the application developers the api. Super simple stuff.

You are missing the point. Stored procedures were not intended to maintain referential integrity of data distributed across boxes.

That is why I was saying that NoSQL is a better solution.

spikesC:
Ofcourse it wouldn't solve every problem but it's a start in the right way if your system permits. And it can get you a huge increase in performance (this article can help explain: https://msdn.microsoft.com/en-us/library/cc505841.aspx or search 'solving database problems with denormalization')

I don't know why you are arguing. Some systems lend themselves easily to distributive systems. My point is that distributing other systems is intractable. I once worked on a migration project where they discovered that they data could not be decoupled. They nearly decided to can the project, which had already cost 10 million dollars. We really had to jump through hoops to migrate the data, and it was a good job that the data was only temporarily distributed for a few months, while it was being migrated.

Geun geun, big big programmer thangs don start.

All these una grammers don big pass me abeg, anyway I don write one small MySQL library like that - for the small boys on the thread to look at - but all these big boys speaking big grammer, na beg i beg o.
They fit come start to dey ask now if my library dey use memcache, dey cache and re-normalize referential integrity - i no sabi the answer of all those ones o.

rushman003:
I'm still a leaner when it's comes to programming, I uses php and mysql, and a perpetual face booker *lol*. But I could help myself whenever I remembered the quantity of users that use Facebook everyday. My little knowledge about database is that the more you have users the slower your site becomes. Even I doubt Oracle could manage 100million users and think of the number of users we have on Facebook. So this challenge is up for pro programmers in the house. What database can you use to manage site like Facebook that has up to 400million users or how do you use mysql to manage it. Do you place this database on different server and users request is sent to a server that has not been saturated?

Facebook stores textual data(status updates , notes etc) in mysql,
and blob data (pictures, videos etc) in one of Apache Cassandra, Hadoop or apache hive (I just know this is the aspect where they use NoSQL the most) . You can check their developer page for more info.

dhtml18:
All these una grammers don big pass me abeg, anyway I don write one small MySQL library like that - for the small boys on the thread to look at - but all these big boys speaking big grammer, na beg i beg o.
They fit come start to dey ask now if my library dey use memcache, dey cache and re-normalize referential integrity - i no sabi the answer of all those ones o.

Hi dhtml, refer.ential integrity basically means that if one record on the database refers to the key of another record, then the other record exists on the database. The database would not let you delete the refer.enced record, for example, while another record refers to it.

For example, imagine you had a table called "PAYMENT", and each PAYMENT referred to a "PAYEE". You wouldn't want to read a PAYMENT record, try to look up the PAYEE, only to find that there is no PAYEE with the given key on the PAYEE table.

But this could easily happen if a program deletes a PAYEE record without checking if there are outstanding PAYMENTS for that PAYEE. It could also happen if a program creates a PAYMENT and a PAYEE record and tries to insert them into the database, but the PAYEE record fails to insert. All manner of poor program design or program bug can mean that the PAYEE is missing. By time you have discovered where the problem is in your program, and fixed the problem, your database may have a lot of missing data. i.e your database lacks "integrity". You can design your database so that it won't let your program make such mistakes. e.g if PAYEE fails to insert, it will throw out PAYMENT, too.

It is very important to learn and know how to use these jargon terms. Programmers remember basic facts and material, but spend a fair amount of time looking up specifications and reference documents only as required. This becomes a key skill, and is only possible if you pay good attention to jargon, because jargon (and acronyms) are used everywhere. Without it, a five minute task of checking some basic fact and behaviour could become a project of a few days.

Brings out my slate and chalk, I must learn all these jargons today today! This is really interesting.
The truth is that all these things, I somehow manage to code them into my apps possibly accidentally.

sinequanon:


You were the one arguing that "NoSQL is not the answer". I think you have lost track of the point.

I was saying WHY NoSQL is useful. It is because it is better suited to TEMPORARY inconsistency and loss of referential integrity.

Referential integrity does not require any joins. You are confusing two concepts.



You are missing the point. Stored procedures were not intended to maintain referential integrity of data distributed across boxes.

That is why I was saying that NoSQL is a better solution.



I don't know why you are arguing. Some systems lend themselves easily to distributive systems. My point is that distributing other systems is intractable. I once worked on a migration project where they discovered that they data could not be decoupled. They nearly decided to can the project, which had already cost 10 million dollars. We really had to jump through hoops to migrate the data, and it was a good job that the data was only temporarily distributed for a few months, while it was being migrated.

We were probably misunderstanding each other, but i get where you're coming from now. I was giving my opinion on how mysql can be performance-tuned, while you're providing a different solution, being the use of NoSql DBs. My bad though

All these big big grammers don break my head finally

Tight ham together dhtml! ....... learning mode activated

It is all over, I give up, I am going back to farming in my village.

dhtml18:
It is all over, I give up, I am going back to farming in my village.

i will dash you my 'hoe ' to take along. bumper harvest in advance...

dhtml18:
It is all over, I give up, I am going back to farming in my village.