Software development and consultant agencies
Studios / Agencies
General Coding Guide
REPL: read–eval–print loop
serverless microservices using AWS Lambda or Google Cloud Functions
Ava and Enzymem and React
Start with designing the schema. You can specify constraints (column size), validation rules, etc. Each time that a column is added or removed, the tool can automatically generate a database migration script. We can also use the constraints and validation rules for both front-end and back-end validation. I got this idea when I was learning meteor.
We should use proper ORM
We should use server side validation
We should implement a mechanism to prevent robot from trashing our database.
Some time ago, I needed to develop a tool that do something with a web page. I needed to do this using Visual Basic, and the interface or library that Microsoft made available for embedding IE inside the Visual Basic application. With VisualBasic, you drag and drop the controls (input fields, or buttons) onto the canvas. VisualBasic automatically generate all the stub methods, and all I had to do was to go down the list, provide implementation (fill in the blank) for the methods that I need to implement. When I reach the end of the list, I know that I am done. Writing software should be this easy.
I also interviewed at a company and their product: you describe the attribute of the objects in your application, and their product generate the application code for you, or something like this. I can't recall this exactly. There is also something called the Wolf Framework, and another thing (look at the mysql-tools page).
Separate front-end development from back-end development using JSON. Front-end developers can work with designer, use stubbed or mocked JSON to create the application. The job of the back-end developers become just "fill in the blanks" (go down the list of stubbed or mocked JSON and implement them one by one). Of course the back-end developers still have to database modeling and all the things that they normally need to do, and front-end developers still have to do all the things that they need to do, but this ability to decouple front-end development from back-end development does give us some freedom, such as front-end developers can start their work while you hire back-end developers. Look into http://hood.ie/
For healthcare and financial application, avoid Java or any language that support multi-threading. Use languages that does not support multi-threading, or run them in a multi-processes manner. The database must be ACID compliance. If the database must use multi-threading, that is fine because database vendors usually test their product for ACID compliance. Think of those HIPPA violations or financial issues that could have been avoided if multi-threading was not used. This does not mean that we are HIPPA-bullet-proof if we are not using multi-threading. There are other requirements that we must implement, but not using multi-threading would save us a lot of time and trouble. If we must use multi-threading, we should use CheckThread or other tools that test for concurrency issues as part of our build / deploy process.
Server push events versus client poll. What are the advantages of server push? What is the impact on the infrastructure (number of servers required, load balancers, firewalls)? Why is server push better than client poll now rather than in the past?
REPL: Read-eval-print loop
Use JSON as the data exchange format. Move application / business logic to the client-side. Use separate library or component for rendering (dealing with different browser or device). The back-end code is to handle server side validation, security, and persistence (storing and retrieving data). I am not sure if this is a good idea (competitor can easily look at the front-end code and re-construct the back-end code), but it seems that some libraries are moving this way. This is food for thought.
Static content should be served using a separate domain, using CDN if practical, compressed and minified.
How can we allow users to choose a different format for the response? Some API users may prefer that we send the response using JSON. Some API users may prefer that we send the response using XML. How can we support this? Which framework offer this capability?
BeanStalk is a private GitHub with deployment
http://vschart.com/compare/tower-js/vs/flatiron/vs/socketstream/vs/derbyjs - An example of how to compare libraries. The row header has a tooltip that explain the meaning. The column header should have position:fixed. Need to add a row for "getting started" tutorial, video tutorials. Why does Node default to MongoDB? Do all apps now run on NoSQL? Need to see if it supports NoSQL. Need to see if it support RDBMS. Data integrity? Download size. Performance benchmark.
http://kafka.apache.org/ - A high-throughput distributed messaging system.
How to store users' private and sensitive data in the database so that the data is not compromised even in the case where our database is compromised or our employees are tempted to see the data?
Database vendors / software do offer features that allow us to store the data encrypted on disk. We should investigate these options first.
Alternatively, we can use something similar to multi-factor authentication. After the user log into the application, we ask the user to enter a PIN. We do not store this PIN in the database. However, we store an hash (MD5, SHA1, or other implementation) of this PIN. After the user provides the PIN, we performs the hash and compare the result with the stored hash string. If the two hash strings equal to each other, the user is validated. We can also use this PIN as the encryption key or decryption key. This way, our employees do not know the key to decrypt the data, unless our employee modify the code to log the PIN to the log file. This approach also has some one draw back. All private and sensitive data has to be serialized as a JSON string before encryption. Perhaps, in our model, we can declare that a particular column requires encryption and decryption, and the model automatically do JSON encoding before encryption, and JSON decoding after decryption.
Supporting the back button for one-page web application:
See the jquery-plugins page and search for History
Miscellaneous Articles and Best Practices:
List of important publications in computer science
Gears and the Mashup problem
Dion Almaer of Ajaxian and Google on Gears, App Engine, and More
5 Things You Need to Know About Raytracing
YQL Geo Library and Introduction to Geo Hacking talk
Tien Tzuo, CTO of salesforce.com spoke on enterprise software
http://www.youtube.com/watch?v=1xUz1fp23TQ&feature=related (Design Patterns, roughly 1 hour)
http://www.youtube.com/watch?v=EEjNb9yUv1k&feature=relmfu (Pattern Matching, 54 minutes)
http://www.youtube.com/watch?v=zZdQ6HXXDiE&feature=relmfu (Software Testing, 55 minutes)
http://www.youtube.com/watch?v=fJaPlsVIzio&feature=relmfu (Lecture 12, Data Modelling - ER Diagrams, Mapping 58 minutes)
http://www.youtube.com/watch?v=WSNqcYqByFk&feature=related (Lecture 11, Data Modelling - ER Diagrams, Mapping, 1 hour)
http://www.youtube.com/watch?v=3VW9ujdB-jM&feature=related (Lecture 41, Case Study - Part One Database Design 53 minutes)
http://www.youtube.com/watch?v=w23fiSVWyM8&feature=related (Data Modeling and Conceptual Sketching in the Design Process, 1 hour 25 minutes)
http://www.youtube.com/watch?v=EUzsy3W4I0g&feature=related (Introduction to Database Management, 53 minutes)
http://www.youtube.com/watch?v=yIFxN8eXfe4&feature=related (The Theory of Normalization, 55 minutes)
http://www.youtube.com/watch?v=C2hSJOS9pks&feature=related (Introduction to UML)
http://www.youtube.com/watch?v=Xxvpwmc-7io&feature=related (The Observer Design Pattern, 7 minutes)
http://www.youtube.com/watch?v=sXLP5SLRrgQ&feature=related (Google I/O 2008 - Design Patterns in an Expressive Language, 44 minutes)
http://www.youtube.com/watch?v=Dtd0njgo4s8&feature=related (10 Things Every Software Architect Should Know, 50 minutes)
http://www.youtube.com/watch?v=gPCUAcbbQ-Q&feature=related (Data Modeling for the Database Developer, Designer & Admin, 58 minutes)
http://www.youtube.com/watch?v=abDEmGtcjkk&feature=relmfu (Cloud Computing, The Next Frontier for OS, 55 minutes)
http://www.youtube.com/watch?v=OFkgSjRnay4&feature=related (Git in One Hour, 1 hour)
http://www.youtube.com/watch?v=Z6-iUNfJsJw&feature=related (Writing LINQ Queries with LINQPad)
Scala versus Node.js:
Static code analysis:
Distributed Management Task Force
Does Java NIO (non-blocking IO) use event?
When implementing a high performance web server, if a request involve a slow IO operation (perhaps it is making another request to another remote resources), we can create a worker thread, which make the remote request and send the result back to the browser. The current web thread can go back to serving other request. See how Netty is implemented.
What make event-driven programming faster than multi-thread programming? What make Node much faster than PHP or Rails or Java?
Event programming with Java.
If event-driven programming does not use threads, then what make Node (which is event-driven) fast?
OOP / OOD
Ruby on Rails
Decision Support Systems
monitoring tools such as Nagios, Scout, Zabbix
automation tools such as puppet, ansible, cuisine, fabric
automation frameworks such as Puppet, Chef
monitoring systems such as Ganglia, Nagios and/or icinga
with chef, cfengine or puppet
automation tools such as puppet, ansible, cuisine, fabric
PHP, Zend, Cake, CodeIgniter, Symfony
Git, Subversion, Perforce
CruiseControl, Apache Ant, Apache Maven, Phing.info
Java, Apache Tomcat, Struts, Hibernate, Doctrine, Spring
Open source SWIG + STAF (Software Testing Automation Framework) staf.sourceforge.net
Google Test Automation Framework
How to type Vietnamese
What is executeScripts=false ?
Apply compiler techniques to iterate at blazing speed
Tornado and Twisted framework (Python)
AMQP messaging protocol
Redis distributed KVS
XPCOM, XPConnect, XPInstall, XPNet
DVD MP3 converter
Optical Pattern Recognition
Natural Language Processing (NLP)
JBI (Java Business Integration)
DDS (Data Distribution Service)
Event Driven Programming versus Multi-threading versus event driven programming combined with multi-threading:
Neural Network / AI / Data Mining / Business Intelligence / Data Visualization:
Data mining / Business Intelligence services:
Domain Driven Design:
Gathering business rules is also called rules harvesting or business rule mining.
Rule Engines for Ruby:
Video presentation on Ruleby
Treetop - You can define your DSL as a Parsing Expression Grammar and then parse it to create your rules in whatever format you like.
Rule Engines for Java:
Rule Engines for PHP:
Rule Engines for .NET:
ActiveRecord vs ORM
MDE / MDA / MDD / UML:
Model-Driven Engineering (MDE) is a software development methodology which focuses on creating and exploiting domain models (abstract representations of the knowledge and activities that govern a particular application domain), rather than on the computing (or algorithmic) concepts. The MDE approach is meant to increase productivity by maximizing compatibility between systems (via re-use of standardized models), simplifying the process of design (via models of recurring design patterns in the application domain), and promoting communication between individuals and teams working on the system.
MDE is considered effective if its models make sense from the point of view of a user that is familiar with the domain, and if they can serve as a basis for implementation systems.
Model-Driven Architecture (MDA) is an alternative to MDE. MDA is similar to MDE. MDA is an initiative from the Object Management Group (OMG).
The main problem with most MDA / MDE tools: the model gets out of sync with the application.
Model Driven Engineering (MDE) -> Unified Modeling Language (UML)
MDA / MDE Tools:
PHP How to generate code from uml class diagram
First create the UML class diagram so that other people can visualize how the system should work. Generate the stub-code. Implement all the methods.
UML tools for Rails:
WSDL (Web Services Description Language):
XML Digital Signature:
Architectural Styles and the Design of Network-based Software Architectures
Building Web Services the REST Way
RESTful Rails Development
A Beginner’s Introduction to HTTP and REST
SOAP vs REST:
RESTful Web Services vs. Big Web Services: Making the Right Architectural Decision
REST for the Rest of Us
SOAP vs REST @IONCANNON
Web Services, Part 1: SOAP vs REST at Ajaxonomy
Giving SOAP a REST by DevX
Roots of REST/SOAP Debate
SOAP was designed as a transport independence, so essentially you should be able to use it over TCP, MSMQ, etc., REST only deals with HTTP
Microsoft's WCF was designed specifically to make SOAP over any transport medium to be as easy as a value in a config file.
SOAP(using WSDL) is a heavy-weight XML standard that is centered around document passing. The advantage with this is that your requests and responses can be very well structured, and can even use a DTD. The downside is it is XML, and is very verbose. However, this is good if two parties need to have a strict contract(say for inter-bank communication). SOAP also lets you layer things like WS-Security on your documents. SOAP is generally transport-agnostic, meaning you don't necessarily need to use HTTP.
REST is very lightweight, and relies upon the HTTP standard to do it's work. It is great to get a useful web service up and running quickly. If you don't need a strict API definition, this is the way to go. Most web services fall into this category. You can version your API so that updates to the API do not break it for people using old versions(as long as they specify a version). REST essentially requires HTTP, and is format-agnostic(meaning you can use XML, JSON, HTML, whatever).
SOAP is good though if you want computers to understand your webservice using a WSDL.
SOAP was designed for a distributed computing environment where as REST was designed for a point to point environment.
Probably the most cited pro for REST is that it is "light-weight" or tends to be more "human readable". At one level this is certainly true, REST does have a lower barrier to entry - there is less required structure than SOAP (though I agree with those who have said that good tooling is largely the answer here - too bad much of the SOAP tooling is pretty dreadful).
Beyond that initial entry cost however, I think the REST impression comes from a combination of the form of the request URLs and the complexity of the data exchanged by most REST services. REST tends to encourage simpler, more human readable request URLs and the data tends to be more digestable as well. To what extent however are these inherent to REST and to what extent are they merely accidental. The simpler URL structure is a direct result of the architecture - but it could be equally well applied to SOAP based services. The more digestable data is more likely to be a result of the lack of any defined structure. This means you'd better keep your data formats simple or you are going to be in for a lot of work. So here SOAP's additional structure, which should be a benefit is actually enabling sloppy design and that sloppy design then gets used as a dig against the technology.
If I were building a distributed, information processing system I would gravitate to SOAP as the communication mechanism (also because of the tranmission and application protocol layering and flexibility that it affords as has been mentioned above).
REST seems more appropriate. AJAX between the client and its server (regardless of payload) is one major example. I don't have much care for the longevity of this type of connection and ease of use and flexibility are at a premimum. Similarly if I needed quick access to some external service and I didn't think I was going to care about the maintainability of the interaction over time (again I'm assuming this is where REST is going to end up costing me more, one way or another), then I might choose REST just so I could get in and out quickly.
Anyway, they are both viable technologies and depending on what tradeoffs you want to make for a given application they can serve you well (or poorly).
If HTTP is available and applicable, use REST. Otherwise use SOAP.
Basically, the Vision & Scope tells you very briefly who needs the software, what their needs are, and how you'll meet those needs (by explaining the features of the software you'll be developing).
How to start a software project?
- Gather the requirements. This is one of the most important step (along with assembling the team, establishing the communication mechanism). When meeting with the client to gather the requirements, your team should include a business analyst, the technical lead / engineer. The business analyst and the technical lead need to have relevant background in the industry for which the product is being built for. Your team may also include a product manager, a project manager, a QA engineer. The more people you can bring to this meeting the better the outcome will be. Everybody bring their own set of questions from their own perspectives so there would be, in the end, less confusion, and less chance for misunderstanding. The trick is each member of your team need to know when to speak and when not to speak. The business analyst should be the one that do the talking. Perhaps, your team can put together a list of questions before the meeting, write each question down on a piece of paper and pass it to the business analyst, submit the question to the meeting forum, and let the business analyst read the question to everyone. Use online meeting provider such as WebEx to communicate your questions. You should record your meeting if possible. Perhaps you should maintain a list of questions that should be asked every time you start a new project.
- Build an online interactive prototype to validate your understanding of the project with the client.
- Decide on what tool will be used for project management so that everyone can see the status of the project.
- Decide on what tool will be used for source code management
- Decide on what tool will be used for Continuous Integration
- Decide on what IDE will be used
- Decide on the processes, tools, server requirements, licensing
- Break it down to smaller features / tasks
- For each requirement, design the database and software concepts and components (what layers, concepts, and components do we need? What tables do we need?). Design class diagrams. Design entity-relationship diagrams.
- Prioritize the tasks
- Design test cases
- Implement just enough code to make the test cases pass
A list of questions to ask the client during the "requirement gathering" meeting / phase:
- What is the purpose of the software?
- What are the required features?
- Who will be the users of the software? The client may not be the end-user of the software.
- How many users will be using the software at any given time?
- How large is the expected user base?
- How frequent will each user use the software?
- What do the users use the software for? How much load will it put on the system?
- What are the user demographic (age, gender, race, religion)
- What data do we need to collect?
- What are the security requirements?
Project Initiation phase starts when the purpose of a new project is defined. The project sponsor identifies:
- Secondary and Primary Goals
- Feasibility Study
- Risk Assessment. Does this project have any potential negative impact on existing business?
- Project Communication Plan
- Resources (team members), and how long (how much time) each team member will be available
- Milestones and timeline for each milestone
- Cost Assessment
- Project budget
The first step in developing a project is drafting a project case study which provides its complete business overview. It defines various project-centric problems and addresses some simple and complex project related issues by identifying a project solution to be implemented.
In "Term of Reference", the pragmatic evaluation of goals, objectives, vision, scope and risks take place. It helps further refining the available resources and structural inputs essential for getting project deliverables.
There is a tendency to over analyze requirements when beginning a project. Requirements, however relevant and appropriate at the threshold stages of a project, have to be modified, made more realistic, and refined as the project progresses.
For the requirement process to be effective, it must take into account:
- The interest of the stakeholder
- The qualification, competence, availability and behavior of all team members
- The size of the project, completion time and budget
- Inherent complexities of the project
The requirement process should comprise of four key factors - total understanding of the project, the step by step project activities, the individual role of all team members, and the communication plan. Everyone on the team should be able to know all the things that are going on with the project.
Brainstorm often: During the initial phases of software growth or even before starting to write a single line of code, you should do several sessions of brain storming. It could be on validating your idea, figuring out competition, predicting the future, picking a programming language, potential learning, etc. Know what you want to build before you build it.
Use good version control system: Even for the most trivial projects, you should try to use version control system.
Document all ideas: When the software is evolving you will have many ideas for new features, doing things differently, or incorporating competing features. Obviously due to lack of resources and time, you won't be able to incorporate all these. But you must document all the ideas, and if possible prioritize them. Keep a single list of ideas. Usually the software will evolve on its own to attract new features. Implement only the most crucial ideas and features, and resist the temptation to add many features.
Include testing and defensive programming: To be successful, sooner or later your software project will need to get out of the demo-mode and face the real world. It might become too late at that time to worry about scalability or glaring bugs if those involve redesigning your software. It saves a lot of time and energy to use common techniques such as good logging, unit testing, performance best practices, and defensive programming from day 1. Also maintain an issue tracker and log even the tiniest of issues with your software. Sooner or later you will need to address them.
Don't wait too long for 1.0 release: In the case of software projects, it is very easy to get started but very difficult to put an end. There is always an endless list of features which needs to be completed before the release, and hence your release never happens. Unless, you make it happen. You will have to make a firm decision about what bugs are important and what can remain as known issues for version 1.0. Implement Minimum Sellable Product.
Be attentive to each team member's needs. We are all human, and we need to socialize, talk to each other. This is what makes our brain grow. All employees are equal (treat them fairly, according to pre-defined order such as how long each have been with the company). Understand them at the personal level, and be supportive of their families' needs as much as possible. Provide an environment that encourage learning, and growth by openly sharing knowledge, cross train, and rotate responsibilities.
A good solution must be fast, easy, and reliable in that specific order. Each of those characteristics is considered ten time more important than the next. That philosophy motivated his team's decisions in almost every area, ranging from visual design to backend engineering.
The thing that people value most is their time. When users encountered flaws in software, they tend to be more forgiving if the software is extremely fast. A crash is less disruptive if the application is quick to restart and offer a quick path back to where the user was.
Speed is especially important because smartphone users tend to have shorter sessions (often less than three minutes long). Users expect applications to deliver relevant information as quickly as possible.
User experience is particularly important. If it takes more than 3 clicks for the user to perform an operation, the users are going to lose patience and easily be drawn away by push notification or other things happening on their phone.
Reliability is also important. Basic robustness and stability are important, but reliability also encompass other ideas like consistency and predictability. Reliable application is one the user can depend on. An application that crash often will annoy the users, and they will probably stop using it.
The API and plug-in architectures are the highest priority. Another way of putting that is: never build anything in-house if you can expose a basic API and get those high-value users (the ones who are getting the most value out of the platform) to build it for you. On the Trello team, any feature that can be provided by a plug-in must be provided by a plug-in. http://www.joelonsoftware.com/items/2012/01/06.html
http://www.jnd.org/dn.mss/why_is_37signals_so_1.html (very opinionated, need to be careful here)
- Being a developer does not stop when you leave your desk in the afternoon — it is a life style. Good ideas usually come when you are not working.
- See the big picture
- Take your time and do it right
- Get a life. Your brain need rest.
- Focus on one thing at a time (without loosing the big picture)
- Be practical