Verification and Evaluation

Go Back: System Attributes Next Step: Design Communication The verification and evaluation stage of the systems engineering process is where we are able to start talking about the current C-BCI options available on the market. To do this we will conduct the usual Verification and Evaluation processes below, but I have also added in a 'feasibility test' for reasons that will become clear later on.

Testing Process
By using our previously constructed Attributes Cascade, we have a set of specific criteria which we can check our design or anyone elses design against. It is conventional to provide the following guidelines for Testing Procedures:

Which Attribute?

 * We will go through all attributes linearly, some can be checked by inspection where as detailed testing is required for others (See example below).

Who Does the Test?

 * Someone with a tertiary level of Electronics knowledge, this is a requirement because they need to check the functionality of the safety systems, which is difficult to do, and paramount for the success of the design.

Procedure Outline

 * Using the Attributes cascade, procede linearly down the tertiary attributes, searching for that feature in the design. For attributes which require more involved testing, refer to the detailed guideline for that particular attribute.

Pass/Fail Criteria

 * For many attributes this is binary and simple to determine, others require almost a subjective determination of the pass/fail critera. Due to the qualification of the Testing technician we encourage them to think about their own benchmarks before referring to the individual guides for each tertiary attribute, where our standards are conveyed.

Attribute

 * Powerful Mathematical filters

Test Person

 * Electrical or Computer Engineer

Procedure Outline

 * 1) Use the device following the steps of the Functional Flow Block Diagram created during the system functions process.
 * 2) While in use record the raw Brainwave patterns for a complete sleep cycle (For all phases).
 * 3) Compare the activity in each wave area to that established in the extensive theory surrounding sleep.
 * 4) A high level of correlation between the results and the theory can be attributed directly to the quality of the data manipulation and noise reduction (mathematical filters), this statement can be made due to the very poor level of raw data received by the few-electrode EEG process used in all C-BCI systems.

Pass/Fail Criteria
As you can see from this example, a failure indicates that the design is not optimised, it is bottlenecked in some way. In this way the Testing process acts as Design Verification.
 * The EEG device is determined to pass if accuracy is over 90%, for results in the 70-90% range, there are still applications available but not suffient ones to justify the design choices.

Evaluation Matrix
Before constructing our Evaluation matrix, we need to decide on a set of properties for our Ideal C-BCI device, these are listed on the dedicated page and are used in the Evaluation Matrix for comparison to other products. Note that none of the products obtain a perfect score, this is of course unavoidable with technological and significant financial constraints in place. The compliance for each customer requirement is determined by looking at the design specifications of our Ideal C-BCI, and the specifications of the three commercially available consumer-BCI devices, which are summarised here.



Discussion
There is a lot of information and Work packed into the above evaluation matrix, as well as some subjective decisions. One major obstacle that was encountered during the Evaluation process was what constituted a 'jump' from a 3 (Full Compliance) to a 5 (Exceeds Compliance). This is particularly evident when considering Our 'Ideal' Design compared to the Emotiv Insight. Due to our increased number of electrodes and emphasis on the software aspects of operation, there is no doubt that our C-BCI design has greater Accuracy and output channels. However it simply can't be awarded a 5 due to the cutting-edge specifications of the g.tec Intendix, which blows the competition out of the water.

So why is the IntendiX not our "Ideal" C-BCI?

This comes down to the way that Customer requirements are formed, and here is where we conduct a feasibility test. Cost is a low-priority customer requirement, and this is not in contention. However if a C-BCI costs too much, a customer simply cannot purchase it, regardless of its feature-set. This is a Go/No-Go criteria. With a price-point over $10,000 the g.tec IntendiX rides the line between Consumer BCI and a medical device. It is impossible to state that the IntendiX can have any success as a device to the average 'Consumer' for the applications that have been previously discussed. The IntendiX is an amazing device for those who need it most, and for an idea of its functionality I recommend viewing the video Here. The IntendiX is where I hope C-BCIs will be in about 20 years. For patients suffering from 'Locked-In Syndrome ' the IntendiX offers a huge increase in Quality of Life.