Unexpected Consequences of becoming a “Technical Tester”

techtesting_title

I want to talk about an experience that has got me thinking in alot more depth about my profession. In my last sprint, I had a story to develop a solution to performance test our system’s midtier.

It was late in the day. It was late in the sprint. I had just finished a complex part of my Gatling script. I needed to write a few lambda equations. These would facilitate a hash map to hold my test data.

I then noticed another problem. I had to fix a dependency in the Gradle build file of my Scala project. This is so that it would point to the correct version of our internal Maven repository. It had recently changed. I made the adjustment and everything seemed to be fine.

I was able to run the code fine on my local machine. I could do the same on my development VM, which is running Ubuntu 14.04. But I also needed to get it working on the teams shared Jenkins instance. This was running on an AWS hosted instance, which had CentOS as the operating system.

More messing around, frantic Googling, and tearing my hair out.  It was 3am and it still wasn’t working, but I had come too far to quit now. I won’t bore you with the details of what I had to go through to get it working. But I got there in the end.

So to get to the point. I had been working solidly on this for days. Even working past the twilight hours and getting up again at 5am to carry on (I was that caught up in it). It was frustrating at times, no doubt. But there is a release of endorphins I get from perseverance and finally solving the problem. This makes it all worth it to me. I learnt a ton in this period, enjoyed my work, and can now look back on it with pride.

Where’s the testing?

But at the end of all this, basking in the glory of all I had learnt and achieved, a thought occurred to me. What has any of this got to do with testing, really? I haven’t done any “testing”, in the traditional sense, for weeks.

I suppose that the final product has testing value. The Gatling script and it’s implementation will performance test our system. But the development work, that I poured many hours and mugs of coffee into, is just that – development work.

The next day, someone outside my Agile team came over to me. “Hey, you’re a tester right?. We need you to manually verify this regression test case. If you could document it with screenshots that would be nice.”

I wasn’t sure what to say. On the one hand, he was right. I am the tester, and this wasn’t an unreasonable request by any means. But I had spent the last couple of weeks immersed in complex and specialised code. To now be thrust into this straightforward and mundane task just felt wrong.

Who else could he ask? You wouldn’t ask a developer to do this. The product owner doesn’t have enough knowledge of how the system works internally. Aside from outsourcing it totally (not practical for a one-off like this), there isn’t anyone else.

This experience isn’t a one-off for me either. It happens more and more. Especially as my technical knowledge and ability grows. I find that I am focusing more on the tools and the code, and want to do more of that stuff.

But this is likely to the detriment of my testing ability. Its the same with all skills, they are like muscles. If you don’t use them, they get weaker over time. Should I be concerned?

What testing should an Agile SDET do?

It got me thinking about how much actual “testing” I do though. I have worked in my current role, as an SDET at SAP, for just over 1 year now. Its a challenging, technical role. As well as writing a lot of code, you need to foster and grow deep domain knowledge.

I absolutely love the role. But I don’t do many of the traditional tasks that I used to previously in my career as a tester. That isn’t necessarily a bad thing, far from it. I don’t write much documentation anymore, other than the occasional test strategy. I rarely write a bug report, I just tell the developer there is a problem. Or I just fix it myself, since I have that ability now.

But shouldn’t I be doing more testing as well? I can’t help but feel that I shouldn’t just be writing code all day. Even if it is code that tests other code. On that note, even if I did just write automation code all day, who tests that code? Do I really think, as an SDET, I can write better code than a regular developer? There’s something to think about…

The rise of the SDET

There is a huge demand in the industry right now for SDETs. Companies want to recruit testers who can write code. They seem to want the perfect hybrid though. Someone with widespread knowledge of testing techniques and when to apply them. But also with the ability to write production quality code.

The problem with this is, it’s hard to find candidates with enough skills on both sides of the equation.


breakit_makeit

Take a typical developer for example. They want to write code. They don’t particularly want to do testing. They certainly don’t want to do manual testing. So they write code to do the testing for them (automation). But automation is not testing!

On the other side, you have the typical or traditional tester. They want to explore the application and find all the ways to break it. Even the ways that nobody else thought of. They don’t just care if the system works or not. They want to know if its effective, if it will satisfy the customer. To think and discover how it can be made better.

These are two significantly different viewpoints. They are the reason that testing and development have been kept distinct in the past. So why are we so desperate to merge them now?

Defining the role of the SDET

The role of the SDET is often not clearly defined. Perhaps that is because it is such a new role in our industry. You can be viewed differently by two people in your company, even two people in the same team.

Should you be picking up all the manual testing? Perhaps some, but I would wager that most SDETs don’t want to do much manual testing, beyond explorartory. These experiences of being an SDET at Microsoft are interesting, and I can relate to them closely at the moment.

Manual testing should be outsourced. Especially when it has to be repeated regularly. It shouldn’t just fall to the pool of SDET resources to pick up. They have their own complex work to do, and its already more difficult that development anyway. Not to mention, an SDET resource is usually paid as much as a development resource. So having the SDETs pick up this work is expensive for the business.

Whats the problem anyway?

Is any of this a problem? Not really. I am discovering that I love writing code, especially when it solves problems. But I also still love testing, finding those unusual ways to break the system. To break the developers (just kidding…) . To think from different points of view.

I’m going to carry on doing both. I think being an SDET is an incredibly exciting and rewarding place to be. There are huge opportunities out there, and they are increasing at a pace.

But I also think its important for companies to remember to make a distinction. Between the traditional tester and the new SDET. They are not the same role, they are different. They shouldn’t recruit a candidate to fill both roles, it won’t end well.

From the perspective of a SDET, I find that it’s easy to get wrapped up in the technical work you are doing. But its important to remember the T in SDET. We aren’t just software developers, we are still testers, it’s in our job title. Its important that we take time to go and explore the system. Find ways to break it. Go back to our roots!

These are some of my sporadic thoughts. I would love to hear your feedback below.

  • http://iTesting.com.au Colin Cherry

    As a Developer who came into Test I can relate to your story from several perspectives even though it was a different era for me with far more bespoke tools – there was almost nothing to buy/acquire so we built our own stuff. Coincidentally, I’m working on a similar thought process at the moment and you have presented some excellent current day challenges that many share grappling with. Excellent post on a growing phenomenon. Good luck.

    • http://james-willett.com James Willett

      Thanks for the feedback Colin – it’s good to hear from someone who has a similar view. I feel this could become a real issue in the industry before long!

  • http://www.sealights.io Eran

    Hi James, great post! I agree. QA is going via a huge transformation, making quality more critical than before. Quality challenges are more complex than before, and require new skills and tools by QA. My thoughts about it : http://bit.ly/2gd8h1E

    • http://james-willett.com James Willett

      Thanks Eran . That’s an excellent piece on your blog, you have raised and identified some interesting points in why we are now inevitably starting to see the convergence of Dev / QA

  • Jim Hazen

    This “As well as writing a lot of code, you need to foster and grow deep domain knowledge.”, this “They seem to want the perfect hybrid…” and this “The role of the SDET is often not clearly defined.” pretty much sum up the new realities of a ‘Technical Tester’. In the past being “technical” meant being oriented more towards the workings of the software and system (and any interfaced hardware to some degree). Technical testers focused on how the software worked from a structural standpoint (i.e., UI to database, application & system API’s, working with printers & plotters & other peripheral devices, or even working with the OS & hardware itself like Disk Utilities). They also worked with the business logic and UI layers (did the correct Stored Procedure get run in the database, did the state of the UI object change correctly, etc.). They even worry about business scenarios to see if it all works together correctly. BUT, it is typically done from a mechanical view (does it do something and does it do it correctly). Other testers may be digging in deeper into the business scenarios to validate the overall process/transaction, is the data on the report correct, did a file get saved correctly to the disk and in the correct format.

    In the past the technical aspects have not been emphasized; mainly due to skill levels and time pressure to just get the testing done along with testing being done later in the schedule. Now we see the shift (finally) towards earlier testing and more technical methods of testing. The SDET is just another form of a tester. Just as Development has people who only do UI work or API work or other specialized work the testing community is finally seeing our own specializations being better defined. I am a Technical Tester; I deal with the mechanics of software testing, tooling for it, and interactions of the software & system. I specialize in Test Automation and Performance Testing, but when needed I use my standard skills to develop Test Plans & Test Cases and other things needed to get the work done. I also deal with other people all day in order to get my work done too.

    So you see, there is a lot we do as Testers anyway in order to get our work done on a daily basis. The toughest part of it all is educating the “others” about what it is we really do, how we do it and why. Just that now we are starting to come out into the light for all to see.

    • http://james-willett.com James Willett

      Thanks for the excellent post and insight Jim. You seem to have a great understanding of your role as a technical tester and the value that you bring to your work. As you say, its now about educating others.

      I have worked for several years as a performance and automation testing consultant, but I am relatively new to the dedicated SDET or Technical Tester role (and I believe there are many like me, since the role is relatively new in most organisations). I think I need to improve my own understanding and definition of the requirements and expectations of the role, so that I can then educate others.

      Great posts like yours really help with this, so thanks again.

      • Calkelpdiver

        James,
        You’ve always been a technical tester by working with Performance & Automation. You have to have technical skills/mindset and employ technical methods to do your work. All you’re seeing is a new “label” on what you have already done. That is why I talk about educating the other people because they think it is something new when in fact it is something that has existed for a very long time but they have not recognized it as such. We as software professionals have an opportunity to improve our standing in this industry, and we better not screw it up.
        Jim Hazen

        • http://james-willett.com James Willett

          Yes, I suppose I do just have a new “label” now of SDET. I think that the difference for me, in the past compared to now, is that as a tester I was always working in a separate team that was somewhat removed from the developers. These days, I am flanked by developers, and hence being so immersed in the agile team has made me feel more “technical”.
          Another great point though. thanks!

  • https://krazycavin.github.io Kevin Cui

    Hi James,

    Very nice you’re sharing your experience. I see the problem of SDET confusion. But I might have a further suggestion regarding to “remember to make a distinction between the traditional tester and the new SDET”.

    First, I’d like to share some thoughts (just my opinions):
    – Testing is a part of development
    Testers do take the responsibility of testing. But I see testing is not a specific job only for testers. For me as a tester, I encourage all team members help testing. Therefore, a part of my duty is to support and motivate team members joining testing. It’s true that the viewpoints of tester and developer are different, but they will come together in order to make product better, right? They’re not opposite to each other. The discussion and collaboration make them support each other.

    – Traditional testers is removable in the team
    Personally, I don’t really like working with bug advocate style testers:
    When they find an edge user case, first they open a bug ticket. Then they will find developers and ask for a must-have fix. If not, the whole release will be blocked.
    Why not discuss the severity with PO first and check with developers the effort for a fix? Not each bug desires to have a fix in real world.
    The role of tester needs to change. Tester should question everything, collect information then provide information correctly and efficiently to people who needs them. (Btw, tester doesn’t try to breaks a thing, tester finds thing which is already broken.)

    – My understanding of SDET
    The definition of SDET sounds like a compromised method to “make testing a part of development” and to “remove traditional testers in the team”. But in my opinions, SDET is a real developer role and a developer I like to work with (who cares about testing :thumbsup:).

    Then back to the situation “Hey, you’re a tester right?. We need you to manually verify this regression test case. If you could document it with screenshots that would be nice.”
    I will ask him back some questions to understand context:
    – Which test cases did you check already? (If not I’d like to pair with him and test together)
    – What could be the potential risk of new changes, not covered yet by test cases? (I prefer to go risk-based testing)
    – Why we need document it? (I won’t do it if it only needs for manager report… as less documentation as possible)

    Finally, my suggestion is to let SDET be developer, a good developer taking care of testing seriously.

    • http://james-willett.com James Willett

      Thanks Kevin, some great points here from you, as usual. Very much agree that quality has to be owned by the entire team, we encourage that approach at our organisation. I also really like your questions to understand the context better, I will remember those for next time!

  • Mauro Pedano (MauGaP)

    Hey, I found your page, and realized there is a lot of information very usefull here!

    I miself am an Automation Tester that is starting its path. Anyway, thanks a lot for sharing your experiences on your page.

    – Mauro Gabriel Pedano

    • http://james-willett.com James Willett

      Thanks for the feedback Mauro

  • Randy Rice

    Great article! I concur with 99.9%. I just disagree with the statement that “manual testing should be outsourced.” There are many reasons not to outsource manual testing, such as the lack of business knowledge, security concerns, etc. Plus, I have consulted in many companies where the test effectiveness of outsourced testers is far lower than that of the in-house testers. These test managers also complain about the lack of creativity on the part of the outsourced testers in defining tests, then the lack of control in how the tests that have been defined in-house are actually performed by the outsourced testers. I know many companies that have pulled testing back into their organization. With that said, yes, there are times when outsourcing some of the testing makes sense – as long as it makes sense in the context of the project and organization. I agree the SDET should be focusing on more technical aspects of testing.

    • http://james-willett.com James Willett

      Thanks for the feedback Randy – I have had similar comments on Twitter from others re: manual testing. I certainly think that there is value in keeping elements of manual testing in house, for the reasons you specified.

      But I find that in my team with our system / product, we have a large number of regression based test cases and house-keeping jobs that are not presently automated. Because we don’t have many dedicated manual testers available, having to execute these test cases often falls to SDETs, even if they have already picked up other sprint work.

      I suppose that the answer for us should be to employ MORE manual testers internally, to pick up this work…. and theres the problem, our company would be unlikely to employ a dedicated manual tester – they would want someone that can demonstrate they can code and be “technical” – but that person is unlikely to do want to be dedicated to manual testing.

      Thanks for getting me thinking about this.

  • Rahul Deshpande

    As Randy Rice and Kevin Cui mentioned, the effectiveness reduces drastically when testing (Automation or manual) is outsourced. Obviously the domain knowledge is missing. There are issues in communication and processes are hard to set. To show the work, the outsourced testers are tempted to raise the defects and then ask questions if any. Also once the project closed, vendor people will go away along with their acquired knowledge, which was transferred by the dev team with extra efforts. And this needs to be repeated for the next vendor (outsourced testing) team.
    Now a days everyone wants to release a product at the earliest by cutting corners. Agile is the concept where most of the projects fails badly due to wrong implementation. As a vendor I worked with a major company from US who thought that breaking a waterfall model in 2 weeks is good to call a sprint. A big boss at my organization mentioned in open forum that manual testing is useless as the developers possess domain knowledge and automation should cover 100% testing. All such wrong concepts are hurting the industry and failing the deliveries. My apologies for straying away from the main topic.

    • http://james-willett.com James Willett

      Hi Rahul, thanks for your insight into outsourcing. I must say that I have heard of similar situations where the outsource company attempts to raise lots of defects just to get the count up and prove that they are providing “value”.
      The more comments I read about the pitfalls of outsourcing, the more negative towards it I am becoming. When I wrote this post I really felt that most manual testing should be outsourced, but I am really starting to backtrack on that thinking now.
      Thanks again for the comment!

      • Rahul Deshpande

        Thanks James for your reply. Yes, there are pitfalls of outsourcing, but by implementing right set of processes and setting the expectations ahead of time, the outsourced team can add value or rather free up the core team from mundane tasks. Communication plays very crucial role here. The client may have different expectations than the top management of the vendor(outsourced) team. This must be handled at top priority by the top management of both companies. Also it is the duty of the client team to make the vendor folks feel at home with regular and open communication. This helps to create free environment making “one team” culture and then there is no need to prove anything to anyone, but to work together delivering the best. I thought of mentioning this so no one should be prejudiced against/towards outsourcing. 🙂

        • http://james-willett.com James Willett

          Thanks for that Rahul, great insight and point again. It seems to me that you have a lot of valuable knowledge and experience in this area, so thank you for sharing!

  • Aaron Evans

    Excellent observations about the role of SDET.

    • http://james-willett.com James Willett

      Thanks Aaron

  • Emsley Dacres

    A really good article James, and as a Test Manager of over 16 years, I’ve seen this same scenario occur multiple times in many companies over the years. As companies push to develop more products faster, the need for testers to implement/develop automated tools, test aid, and environments, to keep pace inevitably creates just the scenario you’re experiencing, and to be honest, it’s a good career progression route for Testers like yourself. However exactly as you say, there’s always the need for hands on manual testing.
    The problem I’ve seen with outsourcing manual testing and the drop in quality often resulting I think comes dow to on of personal investment. People from a third party will rarely feel the same level of commitment and personal investment in a product that the team from the company owning it do, and hence the fall in the level of creative solutions and output that results. This situation can be addressed by creating very close personal ties between organisations, but this isn’t common in my experience and certainly falls outside the remit of Test teams alone to address. It has to be addressed at an organisational level.
    Every organisation I’ve ever worked for that has gone the SDET route has resulted i better more capable Test teams and better products as a result, and they are usually far better able to respond to change effectively.

    • http://james-willett.com James Willett

      Hi Emsley, thanks for the comment and feedback. I love your insight into how outsourcing manual investment leads to a drop in personal investment. That seems so obvious to me when I think about it now, but I don’t think that I had truly considered it before. Its absolutely correct though. I have been reading a lot of psychology books recently and exploring how important it is for employees to feel motivated and invested, if they are to do a good job. It’s about so much more than just money.
      I’m pleased to hear that you have found organisations implementing the SDET route have found success as well. If it’s implemented in an iterative way with constant feedback on what is and isn’t working (i.e. an Agile approach), then I think it will be successful at our organisation as well.

  • Bernhard Burger

    Great article! Having managed QA teams for quite a while (double digit years) I do agree. Especially that relying on test automation makes a lto of people forget that it is only a part of a comprehensive approach to SW quality.
    I also agree that the skillset for SDET is a rare one. But I also believe that, as a manager you should strive to create a diversified team. This is true for gender, background, race, nationality, etc. Because that make the team better. As does a diverse mix in skillsets.
    We understand how brittle and hard to do end-2-end automation is. That is, why we try to limit it, because it is finacially unfeasible. Yet it is necessary to do. And let us not forget what we gain out of the ingenuity and observational capacity of humans. When combined with strong technicians and coders helping the effort one can create a really powerful quality team.
    I do believe testers should understand code and should be able to write basic (test) code. That applies to everyone. But there is room for manual explorations, the designing, the UX testing and the general manual testing as well. And I still see that for the near future. Outsourced: not for, not against, both can work, but you make it look (not sure) that you think about the non-automation testing as second class (not that valuable) work. And I disagree on that.
    One thing on the payment of testers, though. You write that SDET earn the same as developers. Well, good. I’ve made it a point of my career to get my testers (no matter manual or automation) the same salaries as the developers. Because I know that their knowledge and their work is as important and as good as the work of developers. To be honest, I have seen a lot of testers who knew more about the application they tested and the intended customers than any of the developers working on it. And trust me, that is knowledge of incredible value to the development of a product.

    • http://james-willett.com James Willett

      Thanks for the comment Bernhard – I completely agree with your views on automation. We need to remember that its just a tool that helps us to test, we can’t and shouldn’t rely on that to replace our testing! I think I need to convince management of that here – they would like everything to be 100% automated and manually test nothing in an ideal world.

      I find your comment on equal pay for manual testers and developers very interesting. I must say that I have never come across that before, although I think its great that you recognise the importance and value that your testers deliver. I suppose that, in other organisations, it could be more difficult for a manual tester to demonstrate that they have the same value (or greater) than a developer. I mean, a developer can build a great app, and showcase their code. But it might be more difficult for a tester to quantify their value in such a way? Particularly if they aren’t writing more than basic test code?

      Perhaps I should explore ways that testers (manual and automated) can visibly demonstrate and quantify value to their colleagues and management….

      Thanks again for the great insight!

      • Bernhard Burger

        Yes, visibly demonstrating value is harder in manual testing. And therefore it is hard to convince management, if they haven’t worked in that area as well. What I find helpful is asking the developers in the teams, what they think about the value of the tester’s work. Most of the time they will definitely know and tell you about the value of “their” testers and all other roles within the process. Think about product owners, scrum masters, BAs, they all play a role, and it is hard for them to show their work in a “countable” way as well. Even though we are in an engineering field, we should not forget we deal with people – and people are not described by “numbers” or “things to show” only, as we all know.