Welcome, Guest: Register On Nairaland / LOGIN! / Trending / Recent / New
Stats: 3,207,714 members, 8,000,026 topics. Date: Monday, 11 November 2024 at 07:17 PM

Solving Real Problems - Programming (2) - Nairaland

Nairaland Forum / Science/Technology / Programming / Solving Real Problems (6357 Views)

Solving The Nigerian Budget Padding Issue With PHP (2) (3) (4)

(1) (2) (Reply) (Go Down)

Re: Solving Real Problems by seunthomas: 2:39pm On Mar 09, 2016
Imaginery company HackALot wants to hire you develop a monitoring system for the government of XYZ country. The country has been getting a lot of terrorist threats recently and recruited the company to build a system to monitor phone conversation and internet traffic for anyone in the country. HackALot already has access to the data and only needs to develop a system to monitor and detect anomalies that can lead them to determine if a specific person poses a terrorist threat or not.
Using what ever technology possible, explain how you would go about building this kind of system.
Major consideration is that the system must work in realtime.
Assumptions all phone call data has been converted to text representation, encrypted internet traffic has been decrypted to plain text.
Re: Solving Real Problems by Nobody: 12:05am On Mar 10, 2016
huh
Re: Solving Real Problems by larisoft: 9:53pm On Apr 09, 2016
uhmmm...let me try

I d have a listener triggered every time a new batch of data is received,make objects of these data and pass the objects, each with fields like type, content, sender, receiver etc, as parameters to a 'process' method.


in this method, using several objects, algorithms and libs, I ll search for occurrences of terrorist-like words in the data like " bombs, explode , target. Match pictures to faces of known terrorists. match recordings to vocals of radical sentences. and grade each object based on the frequency by which these words/pictures/audio occur before inserting into db and notifying listeners that new information is ready.


at the front end, an Ajax call simply monitors one of these listeners every 5 or so seconds (could be a txt file containing the no of updates made to db so far) once it detects a change, it queries the.database and refreshes the screen with new results of the highest threats.

pretty much all technologies can be used but java and c# could be favored since this system will require a lot of processing power that ll be missed in interpreted languages...

ngwa, correction, sir.

1 Like 1 Share

Re: Solving Real Problems by asalimpo(m): 10:30pm On Apr 09, 2016
A system like this would certainly be very hard to create because of the human element.
Humans are far smarter than machines.
Say, the machine succeeded and a terrorist cell was brought down.
The other cells will change their tactics and beat the system hands down.
e.g

they'll communicate codedly, such that their communication will go under the snoopers noses undetected.
imagine this kind of conversation:
A : Man, i saw a fine shoe i like but it was very expensive?
B : How much?
A : $1000
B: why didnt you borrow money to buy it
A : The bank wouldnt lend me?
B: ugh, banks are no good, next time, go to a lender
A : dyu know any?
B : Yes,
A : where?
B: i know one at 18 broad street, and another at 20 minks street
A : okay, i'll check them out
B: ok, tell me how the next bargain goes.

How would you suspect a terrorist activity from such a conversation?!
But this could be terrorist conversation transpiring codedly!

for those still confused here's a lil of the top of my head encoding.

"i saw a fine <whatever> " - I've found a suitable target but
"it was very expensive " - the security detail is too much
"why didnt you go to banks " - will the usual avenue not work?
"bank's wouldnt lend me " - no, brute force or usual tactic will fail
"go to the dealers " - meet master strategists, higher ups, to pull strings.
"18 broad street, mink street " - useless detail that doesnt mean anything , meant
to confuse any eaves dropper.
the other person knows who the "dealers" are.

1 Like

Re: Solving Real Problems by larisoft: 10:15am On Apr 10, 2016
asalimpo:
A system like this would certainly be very hard to create because of the human element.
Humans are far smarter than machines.
Say, the machine succeeded and a terrorist cell was brought down.
The other cells will change their tactics and beat the system hands down.
e.g

they'll communicate codedly, such that their communication will go under the snoopers noses undetected.
imagine this kind of conversation:
A : Man, i saw a fine shoe i like but it was very expensive?
B : How much?
A : $1000
B: why didnt you borrow money to buy it
A : The bank wouldnt lend me?
B: ugh, banks are no good, next time, go to a lender
A : dyu know any?
B : Yes,
A : where?
B: i know one at 18 broad street, and another at 20 minks street
A : okay, i'll check them out
B: ok, tell me how the next bargain goes.

How would you suspect a terrorist activity from such a conversation?!
But this could be terrorist conversation transpiring codedly!

for those still confused here's a lil of the top of my head encoding.

"i saw a fine <whatever> " - I've found a suitable target but
"it was very expensive " - the security detail is too much
"why didnt you go to banks " - will the usual avenue not work?
"bank's wouldnt lend me " - no, brute force or usual tactic will fail
"go to the dealers " - meet master strategists, higher ups, to pull strings.
"18 broad street, mink street " - useless detail that doesnt mean anything , meant
to confuse any eaves dropper.
the other person knows who the "dealers" are.

for computers to be bought into the equation, we have to assume that relevant data is already available and need to be processed into information. how these data are gotten is not our businesses but that of the law enforcement agents
Re: Solving Real Problems by seunthomas: 12:57pm On Apr 10, 2016
larisoft:


for computers to be bought into the equation, we have to assume that relevant data is already available and need to be processed into information. how these data are gotten is not our businesses but that of the law enforcement agents
In addition to your contribution. Most agencies employ human operatives to gather information(sigint). The information they gather is fed into the system we are proposing in addition to data that is collected. Although this systems are very complex, they already exist around the world. One such system exist in Nigeria (online gossip).
Re: Solving Real Problems by Nobody: 1:54pm On Apr 10, 2016
larisoft:
uhmmm...let me try

I d have a listener triggered every time a new batch of data is received,make objects of these data and pass the objects, each with fields like type, content, sender, receiver etc, as parameters to a 'process' method.


in this method, using several objects, algorithms and libs, I ll search for occurrences of terrorist-like words in the data like " bombs, explode , target. Match pictures to faces of known terrorists. match recordings to vocals of radical sentences. and grade each object based on the frequency by which these words/pictures/audio occur before inserting into db and notifying listeners that new information is ready.


at the front end, an Ajax call simply monitors one of these listeners every 5 or so seconds (could be a txt file containing the no of updates made to db so far) once it detects a change, it queries the.database and refreshes the screen with new results of the highest threats.

pretty much all technologies can be used but java and c# could be favored since this system will require a lot of processing power that ll be missed in interpreted languages...

ngwa, correction, sir.


Your Algorithm would be too slow because no filter to monitor phone calls , text of millions and Internet activities all at once . One hour may crash the system .

This is a very big project that requires database centres like Google and Facebook . One terabyte per minute is not an easy setup .
Re: Solving Real Problems by Nobody: 2:00pm On Apr 10, 2016
proxy20:



your Algorithm would be too slow because no filter to monitor phone calls , text of millions and Internet activities all at once . One hour may crash the system . Your super computer processor may take days to process querys , it will load for days with no hit .

This is a very big project that requires database centres like Google and Facebook . One terabyte per minute is not an easy setup .
Re: Solving Real Problems by seunthomas: 4:39pm On Apr 10, 2016
[quote author=proxy20 post=44576741][/quote]
I dont think it would be slow in the age of c1000k and c10000k systems.
Re: Solving Real Problems by seunthomas: 4:40pm On Apr 10, 2016
proxy20:



Your Algorithm would be too slow because no filter to monitor phone calls , text of millions and Internet activities all at once . One hour may crash the system .

This is a very big project that requires database centres like Google and Facebook . One terabyte per minute is not an easy setup .
I dont think it would be slow in the age of c1000k and c10000k systems.
Re: Solving Real Problems by larisoft: 7:44pm On Apr 10, 2016
[quote author=proxy20 post=44576741][/quote]

We are talking optimized algorithms here. Besides, any one asking you to process images and recordings of an entire nation probably knows or should know what they are up against.

I'd be thrilled to hear better solutions though.
Re: Solving Real Problems by asalimpo(m): 7:56pm On Apr 10, 2016
Object oriented programming , in this situation just won't cut it.
Creating all those millions of objects per millisec or second, then tearing them down to process
requests is a recipe for disaster. Not to talk of using the now-dated threading model of OOP languages at the server side.

A more radical approach , like high fault-tolerance,concurrent systems /architecture is what i thnk would do,with batch mode processing of data, instead of polling (using a listener).
This is where massive parallel processing and purpose built languages (erlang? ) and datastores (hadoop like stuffs) would shine.
fault-tolerance,
distributed,
parallel processing

keywords to watch for , for projects like this.

like some1 said previously, this isnt something for solo coder working at his machine but for larger machines : dedicated data centers/mainframes .
Re: Solving Real Problems by asalimpo(m): 7:56pm On Apr 10, 2016
--
Re: Solving Real Problems by gojishi: 1:39pm On Apr 11, 2016
seunthomas:

They told u they use it in production right
Hi Bro,

FYI, Node.js scales very well - cluster are used to scale vertically and horizontal scaling can be effected using tools like RabbitMQ as well as the UDP module to handle messaging. And FYI companies like Paypal, Netflix and Linkedin among others do use it in production.

(1) (2) (Reply)

Which Do I Learn First C Or C++ / I Want To Become A Good Systems Programmer Before I Graduate / Help A Frustrated Programmer With Power Problem

(Go Up)

Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health
religion celebs tv-movies music-radio literature webmasters programming techmarket

Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 33
Disclaimer: Every Nairaland member is solely responsible for anything that he/she posts or uploads on Nairaland.