NHS Medical Careers - Report 2010

Embedded Scribd iPaper - Requires Javascript and Flash Player
An evaluation of the NHS Medical Careers website
Robin Mellors-Bourne CRAC - Careers Research & Advisory Centre
For the Kent, Surrey and Sussex Postgraduate Deanery November 2010
Evaluation of the NHS Medical Careers website
Acknowledgements
First, I am grateful to all the students, trainees, doctors and education/career professionals who gave their time to respond to the surveys and/or take part in interviews, without whose input this study would have been impossible. Particular credit and thanks are due to all the individuals working in medical schools and deaneries who forwarded our e-mail invitations to their students and trainees, and to Amy Stringer of the Medical Schools Council who helped with this process. Lisa Stone and Jason Yarrow of the KSS careers team provided enthusiastic and willing assistance at various points within the research, which was invaluable. Analysis of the relatively complicated results dataset was only possible with the expertise of my CRAC colleague Goska Leslie.
Careers Research and Advisory Centre (CRAC) Ltd Sheraton House Castle Park Cambridge CB3 0AX Tel: 01223 460277 enquiries@crac.org.uk www.crac.org.uk © Careers Research & Advisory Centre (CRAC) Ltd. 2010
ii
Evaluation of the NHS Medical Careers website
CONTENTS
1. 2. 3. 4. EXECUTIVE SUMMARY BACKGROUND AIMS AND OBJECTIVES OF THE RESEARCH METHODOLOGY AND RESPONSE SAMPLES
4.1 4.2 4.3 4.4 4.5 Conceptual model Online survey methodology Online survey response sample In-depth interviews Data presentation
1 3 6 7
7 8 8 9 9
5.
RESEARCH RESULTS
5.1 5.2 Respondent characteristics 5.1.1 5.2.1 5.2.2 5.2.3 5.3 5.3.1 5.3.2 5.3.3 5.3.4 5.4 5.5 5.4.1 5.5.1 5.5.2 5.5.3 5.6 5.6.1 5.6.2 5.7 Implications of the samples obtained Awareness amongst potential users Frequency of use of the website Summary and implications Reaching the site Motivations to use the website Users’ engagement in career thinking Implications Observations Usage of main content sections Media-rich content and interactive functionality Summary of assessments of content Survey responses Insights from interviews Awareness and use of the website amongst target audiences
10
10 12 14 14 16 17 19 19 20 21 22 24 25 27 27 29 32 34 34 35 36
How and why users came to the website
Impressions of the website and its offer Detailed usage and usefulness of content
Views from professionals
Views from non-users
iii
Evaluation of the NHS Medical Careers website
5.8
Measures of impact 5.8.1 5.8.2 5.8.3 Users’ perceptions of value and impact Derived impact measures Summary of evidence for impact
38 38 42 44
6.
SUMMARY OF FINDINGS AND RECOMMENDATIONS
6.1 Overall findings 6.1.1 6.1.2 6.1.3 6.2 Levels of awareness and use Detailed utilisation and usefulness of content Value and impact of the website
45
45 45 46 47 48
Recommendations
7.
REFERENCES
50
iv
Evaluation of the NHS Medical Careers website
1.
1.1
EXECUTIVE SUMMARY
Research objective and methodology
Independent research has been carried out to evaluate the NHS Medical Careers website, which aims to be an online ‘first port of call’ for those in medical training and their advisers who have a question relating to medical careers. The study sought evidence from users and potential users in relation to the awareness, utilisation and possible impact of this website amongst its target audiences of medical students, postgraduate doctors on Foundation and Specialty Training Programmes, and the education professionals who support them. An online survey of potential users measured levels of awareness and use. Responses from those who had used the website were combined with responses to a parallel survey of registered users in order to investigate detailed utilisation and potential impact. The 1450 responses achieved were supplemented by interviews with careers/education professionals. 1.2 Awareness and use of the website
There appears to be considerable awareness of the website amongst its main target audiences – approaching half of medical students and those in Foundation and Specialty Training Programmes know of its existence, with the highest levels of awareness (over 50%) amongst Foundation trainees. Around 40% overall of these ‘potential users’ had actually used the website at least once, and nearly 60% of Foundation Year 2 trainees. Awareness and use by its target audience was observed across all four nations of the UK and all the English regions. Uneven results of sampling prevented full analysis of variations in awareness or use by individual region, but relatively even levels of awareness and use were recorded across a diversity of English regions and Scotland. Awareness and use are highest amongst Foundation Programme trainees, but the levels of awareness and use amongst medical students and especially Specialty trainees are significant and arguably impressive given the relatively infancy of the website. Most potential users who had not been aware of the website seem enthusiastic to use it in future. Although most users (especially students) reach the site via online search or a link from another website, substantial numbers are referred to it by careers/educational advisers, suggesting that the site is well embedded. Principal motivations to use the site are to gain general medical careers awareness or specific knowledge to support a decision or application. Few users had benefited from a formal careers interview recently, which
1
Evaluation of the NHS Medical Careers website
endorses the need for online careers information to underpin more widespread informal discussions and self-guided learning. 1.3 Website utilisation and usefulness of content
Most users made repeat visits to the site, perhaps reflecting their positive views of its content, presentation and organisation. They appear highly satisfied with the range, depth and currency of the content, almost all of which they found useful. The website’s content relating to specialty information, career options and career planning was most widely used, but targeted content for specific audiences was widely viewed by and useful to those audiences. Almost no content was deemed to be of low value. Particular features of the site are video content and also interactive online career decision support tools. Video content, mainly in relation to specialties, had been viewed by 1 in 6 users, and 1 in 3 of frequent users, and its usefulness highly rated. The interactive tools had been used by 1 in 4 users and were also rated as very useful by most. Advisers consider these content features to differentiate the site as unique from other online sources. Although overwhelmingly users appear satisfied with the range and nature of content, there were recommendations for additional content in relation to specialties and ‘interpreted’ workforce statistics (to understand competition or success ratios). 1.4 Impact and value of the website
There was unequivocal quantitative evidence, from trainees and students, that the majority perceive the NHS Medical Careers website to be valuable to them. Medical careers/education professionals have strongly positive views about the site’s value; the vast majority reported it ‘invaluable to them professionally’ and all thought it valuable to trainees and students, and that more would use it if they knew about it. They believed the site contained information that was not available elsewhere, and did provide an invaluable ‘first port of call’, from which there were links to even more detailed information if needed. They perceived the site to be unique in offering detailed information in a non-judgemental and user-friendly manner and high value content assets such as video and interactive tools. There was also some evidence for correlation of high levels of some career-related activities and attitudes, indicative of greater engagement in career thinking, and more frequent use of the website. While not providing independent ‘proof’ of impact, this reinforces the perceptions from users and professionals that the website has a positive impact on its target audiences and should be continued as a discrete element of the online careers support landscape.
2
Evaluation of the NHS Medical Careers website
2.
BACKGROUND
The Modernising Medical Careers (MMC) programme was initiated by the Department of Health with the aim of improving patient care by delivery of a modernised and focused career structure for doctors through an extensive reform of postgraduate medical education (Department of Health, 2004). The reforms introduced a new progression pathway which included the Foundation Programme and a range of Specialty Training Programmes. One of the 18 key principles which underpin the MMC programme states: ‘Rigorous counselling and career advice should be available through training’. The MMC Working Group for Career Management has the view that provision of support for career management needs to be an integral part of medical education and human resource management (MMC Working Group, 2005). This builds upon prior recognition that doctors in the Foundation Programme need access to high quality assistance with planning their careers (Department of Health, 2003). One implication of the MMC Programme structure is that Specialty training now has more defined starting and end points, as a result of which ‘directional’ choice needs to be made somewhat earlier by trainees; as a result they may have potentially less opportunity to experience, and discuss informally, differing specialty work prior to that choice. Interest in the provision of career support to medical students and doctors continues. Professor Sir John Tooke’s inquiry into the MMC programme (Tooke, 2007) included in its Recommendation 17 the importance of accurate data on specialty competition and workforce statistics to underpin career aspirations and choices, and that medical schools should play a greater role in providing careers advice. Summarising a series of cohort studies, and specifically how eventual specialty relates to earlier career intentions, Goldacre et al. (2010) have recently demonstrated that many doctors experience changes in direction, reflecting that not all make effective long-term specialty choices early in their training, and that others would benefit from greater flexibility. In the context of MMC, the implication is potentially a greater need for effective career-related information and advice to support Specialty training decisions in particular. Further, the Department of Health’s review leading to the Darzi report (Darzi, 2008) identified the need for ‘a clear focus on improving the quality of NHS education and training’. As recommendations from the Tooke and Darzi reports are implemented, resulting changes to medical career pathways continue to offer challenges to the provision of careers information, advice and guidance to doctors, especially those in training. A recent evaluation of the Foundation Programme (Collins, 2010) confirms: ‘Trainees need accurate information regarding future opportunities in various medical careers, and a realistic
3
Evaluation of the NHS Medical Careers website
idea of where their career path is likely to lead. The delivery model for healthcare is changing, and trainees need to be clearer about the needs of the NHS and the wider health economy. There is [also] a broad consensus that careers information and guidance should be provided early in a student’s career, including prior to medical school.’ Research has been carried out into career choices and progression for doctors within cohort studies of 1995-2005 medical graduates by the British Medical Association (BMA) and the UK Medical Careers Group. The BMA’s report ‘Signposting Medical Careers for Doctors’ (BMA, 2003) made eight key recommendations, of which one stated ‘further research is needed to test the effectiveness of the various methods of delivering careers advice’. Independent research carried out by Jackson et al. (2003) has also identified the need to equip doctors in medical training to manage their own careers through the growth of interventions to: ‘develop career management skills, understand their interests and appraise their strengths and weaknesses and develop action plans for their career development and make more informed career decisions’. The report also concluded that ‘this new approach needs to be backed up by the development of self-help (e.g. web based) career materials for doctors’. Collins’ evaluation considers that there is now extensive provision of careers information and advice, but that the particular challenge remaining is to help trainees to manage their career expectations against the realistic opportunities available to them (Collins, 2010). It recommends that providers should work together to define good practice in careers information and advice provision and that the key need going forward is to provide easily accessible, simple to understand information which contains data on each specialty including competition ratios and likelihood of success. Medical schools and postgraduate deaneries have a key role in delivering careers information to medical students and trainee doctors, respectively, formally within their education programmes but also in the form of advice and guidance, both face-to-face and online. Patel et al. (2008) have highlighted the importance of online tools and resources, identifying that ‘medical students prefer computer-based technology and rapidly adapt to new resources on the web because of its availability, ease of use and speed in retrieving relevant information’. As alluded to by Collins, there is now a wide variety of online provision of career support and information through individual postgraduate deanery websites, medical school websites, Royal College websites, as well as national offerings such as NHS Careers, BMJ Careers and the MMC site itself. Although there is much detailed information available across this
4
Evaluation of the NHS Medical Careers website
plethora of websites, the NHS Medical Careers website was launched in November 2008 with the aim of providing a central, up-to-date source of information to support the career decision-making of medical students and postgraduate doctors in training. The website is currently managed by the careers team of the Kent, Surrey and Sussex Deanery under contract from the Department of Health, with the support of an advisory group which includes representatives of key stakeholders. The vision for the development of the new website was that, within two years, it would be seen and used as a first step by medical students and postgraduate trainee doctors, as well as medical education trainers and careers specialists, who had a careers question relating to working as a doctor. Subsequent to its original launch, based on feedback recorded on the website, a series of recommended improvements (summarised in Hallam, 2009) were incorporated into a revised site design prior to a re-launch in July 2009. These included a range of materials developed and licensed from the Association of American Medical Colleges (AAMC) Careers in Medicine programme.
5
Evaluation of the NHS Medical Careers website
3.
AIMS AND OBJECTIVES OF THE RESEARCH
The principal aim of this research study was to assess the degree to which the NHS Medical Careers website is fulfilling its role as a first online port of call for medical students, postgraduate doctors, trainers and career specialists, who have a careers question relating to working as a doctor. However, it was thought desirable for an external evaluation of the website – from the perspective of the core target user groups – also to contribute a needsbased input towards further development of the website and to the overall body of knowledge in relation to career support in medicine. In meeting the principal aim, there were several research objectives: • To evaluate the current ‘visibility’ of the website: e.g. how many people are aware of it, how accessible it is and whether it adequately covers all four UK nations; • To evaluate the utilisation of the website, particularly the ‘fit’ between the information, tools and resources provided on the site with the needs of the core target groups, and to identify any gaps in its current provision; • To identify quantifiable impact and benefits of the site to individuals seeking career information, and especially in relation to the medical specialties: essentially, does the site effectively improve the lot of medical students and doctors in training? • To explore the value of different ways in which information, tools and resources on the site are presented, in terms of being attractive and accessible to the target groups, including the benefit of using media-rich content and interactive online functionality; in addition to explore what else might be provided and how the site might be developed to support medical career choice more effectively. The main target user groups were agreed to be medical students, postgraduate doctors in Foundation Programme and Specialty Training Programmes, together with medical education trainers and professionals, and other professionals offering careers advice/support to those groups.
6
Evaluation of the NHS Medical Careers website
4.
4.1
METHODOLOGY AND RESPONSE SAMPLES
Conceptual model
Conceptually, the main research objectives concern measures of visibility, utilisation and impact of the website amongst its target audiences: • “Visibility” – Awareness of the website amongst potential users and accessibility to them • “Utilisation” – Assessment of the usage of the site by different user groups and how they experience it in relation to their perceived needs • “Impact” – Identification of the consequent effects on and benefits to users.
These could potentially be addressed by a series of research approaches and instruments (Figure 4.1). In order to obtain a full picture across all three measures, particularly in respect of career-related impacts on users, research would be necessary over a far longer period than was available for this commissioned study. Within the time and funding constraints, it was therefore necessary to maximise the range of information obtained from each research instrument used, across the three objectives. Data and information from the instruments used and supporting dialogues would then be used together to build a composite set of evidence. Figure 4.1 Conceptual model
Objective
Possible research approaches
Awareness surveys
Visibility
User surveys
Website statistics User dialogues User feedback User engagement Stakeholder and 3rd party dialogues Longitudinal user impact studies
Utilisation
Stakeholder dialogues
Impact
7
Evaluation of the NHS Medical Careers website
The research study was therefore undertaken using a mixed method combining quantitative surveys of the medical student and postgraduate trainee target groups, backed up by telephone interviews with selected participants to provide additional, in-depth understanding and contextual information. This new primary research would be cross-referenced to existing website statistical evidence and feedback where possible.
4.2
Online survey methodology
In order to obtain information from students and trainees, a quantitative survey approach was used. An online questionnaire was created to collect responses, dominantly using closed questions to obtain information but with a number of open-ended questions inviting comments on particular issues. Different groups of questions were provided for different subsets of respondents, for example according to whether they had used the website or not, and some bespoke questions were offered to medical education professionals who responded to the survey. The maximum number of questions that could be answered was 28, and testing showed that respondents took 15-20 minutes to complete the questionnaire. The main attraction strategy was the issue of e-mails to different groups of users and potential users, inviting participation in the research, via a hyperlink to the online survey site, offering a prize draw as an incentive. In addition, a hyperlink to the online questionnaire was placed on the homepage of the website for the duration of the research, and highlighted within promotional material for the website. Two main e-mail campaigns were undertaken. The first was an e-mail issued to all those who had registered with the website prior to commencement of the project, which then comprised 3435 registrants. This was issued on 30 June 2010 and repeated a fortnight later. A repeat emailing was also made in September 2010. The second strategy was a wider attraction campaign using contacts within a variety of medical schools and postgraduate deaneries who were asked to forward the e-mail to their students and trainees respectively, in order to canvas views more widely. This second campaign took place during October 2010, with closure of the online survey on 9 November 2010.
4.3
Online survey response sample
In total, 1504 responses were received to the online survey, of which 484 were in response to the e-mail issued to registered users. The latter group represented a response rate of approximately 14%. From e-mail system data, it was known that roughly 30% of registered users had opened the invitation e-mail, so the ‘click through’ rate to complete the survey was nearly half of those who had opened the e-mail. For this particular group within the sample,
8
Evaluation of the NHS Medical Careers website
the response level indicates a confidence interval of around 3-4% at 95% confidence level (i.e. 95% of cases should fall within a c.3-4% ‘error bar’ for a particular data point). For the remaining respondents, it is not possible to calculate a response rate as the total number invited to participate (by a variety of contacts) is unknown, and a small proportion of other responses (around 5%) were from site users who had clicked through to the survey rather than being invited by e-mail. From the 1504 gross responses, duplicates and very incomplete responses were removed prior to analysis, leaving 1459 responses which were either complete or substantially complete as the final overall sample, on which analysis was conducted and findings from which are presented in this report.
4.4
In-depth interviews
To understand the experiences and perceptions of a selection of respondents in more detail, a modest number of semi-structured interviews were carried out by telephone. These were selected from respondents to the online survey who had indicated that they would be prepared to undertake an interview, including a range of respondent types. The findings are largely presented as corroborative insights, or as contrasting views, along with the results from the survey, supported by some verbatim quotations.
4.5
Data presentation
Data from the online survey are presented in the remainder of this report in tabular and graphical format, together with sample response sizes (N) and descriptions. All data are unweighted. Percentages have been rounded to the nearest integer; due to rounding, the totals of a column or row in a table may not always sum to 100%. Percentage figures below 1% are shown as *%.
9
Evaluation of the NHS Medical Careers website
5.
5.1
RESEARCH RESULTS
Respondent characteristics
The responses comprised two sub-samples, one intended to be representative of registered users of the website (‘registrants’) and the other a purposive sample of ‘potential users’ (some of whom would have used the site, but many who would not have). As a result of these different types of sub-sample, analysis of the respondents overall as a single group is of limited value. However, the training stage of respondents within the overall sample is shown in Table 5.1, along with the same information for each of the sub-samples. The proportions in each training stage within the two sub-samples were broadly similar, with rather more students and rather fewer Foundation trainees in the ‘potential user’ sub-sample. Of those who were undertaking medical training, which was about 9 out of 10 respondents, approximately half were medical students, slightly under a quarter in the Foundation Programme, and about a quarter in Specialty Training, along with a small percentage of qualified doctors and some undertaking training outside the UK who did not ‘fit’ the stage options offered. The remainder were medical education/training professionals and careers advisers, and a small number of people considering training in medicine. Table 5.1 Survey respondents by training stage or other type, overall and for each of the two sub-samples. Figures in parentheses are recalculated to show proportions of those within medical training
% of all respondents Medical students Foundation Y1 trainees Foundation Y2 trainees Specialty trainees Other Education professionals ‘Considering medicine’ Count (N) 42 10 13 23 3 8 2 1459
% of ‘potential users’ 47 (51) 9 (9) 10 (11) 25 (27) 3 (3) 6 1 975
% of registered users 34 (40) 13 (15) 15 (18) 20 (23) 3 (3) 11 3 484
10
Evaluation of the NHS Medical Careers website
Information on key demographic characteristics of respondents was obtained, including data on gender, age, ethnicity and original domicile/nationality. However, potentially the most significant issue for this study is the working or training location of respondents, as it could reveal the extent of awareness and use of the website across its target regions and beyond. Table 5.2 records the work or study location of survey respondents, based on free-text responses which were then coded to nations and/or regions. This proved to be quite a challenging analytical task, as respondents did not refer consistently to their institutions. Many named individual hospitals or NHS Trusts, or other institutions, rather than their respective postgraduate deaneries, as well as universities and medical schools, and 6% provided insufficient information for analysis. However, the majority were successfully identified and coded, revealing that: • Responses were received from students, trainees and doctors in all the English regions, with highest response numbers in the North West; • A large number of responses was received from Scotland and a modest number from Wales, but very few from Northern Ireland; • There were responses from students in every GMC-accredited undergraduate medical school in the UK other than Aberdeen and Belfast; • There were responses from postgraduate trainees and doctors who were identifiably from all UK postgraduate deaneries with the exception of Defence and Northern Ireland; • At least 36 responses came from overseas, mostly from Asia but a few from eastern Europe, probably reflecting ‘casual’ site users who had clicked through to the survey. The geographical distribution of responses in the two sub-samples were markedly different, as demonstrated in the two right hand columns of Table 5.2. There were responses from registered users in all regions and UK nations, but with highest numbers in London and the South East, and modest numbers in the South West and in Scotland. On the other hand there were most responses from potential users in Scotland and the North West, a modest number in the South East but relatively few in other regions; this strongly reflects the varying and somewhat unpredictable participation of contacts in different regions to forward e-mails to potential respondents. The different distributions within the two sub-samples do have some implications for certain aspects of analysis, as will be seen.
11
Evaluation of the NHS Medical Careers website
Table 5.2 Location of respondents’ place of work, training or study, by nation and English region, for overall sample, ‘potential users’ and site registrants
Nation England London South East South West East of England East Midlands West Midlands Yorks & Humber North West North East Scotland Wales Northern Ireland Other nations Uncertain Count (N) 113 156 57 54 53 75 21 322 39 351 59 61 36 93 1459 Region No. of (total) respondents % of (total) respondents 69 8 11 5 4 4 5 1 22 3 24 4 * 3 6 % of ‘potential users’ 56 4 10 3 4 2 4 * 30 3 32 5 * 1 3 % of registrants 75 18 14 10 4 7 8 5 7 2 10 3 * 6 6
Analysis and reporting in the remainder is report is based either on the ‘potential users’ subsample (chiefly for issues relating to awareness and use) or on users of the website (i.e. a combination of all the ‘registered users’ sub-sample and a proportion of the ‘potential users’ sub-sample).
5.1.1 •
Implications of the samples obtained Irrespective of any issues around the potential ‘representativeness’ of either subsample of respondents, significant numbers of responses were achieved from all the website’s target audiences in terms of stage or role in medical education and training).
12
Evaluation of the NHS Medical Careers website
•
The geographical distribution of potential user respondents strongly reflects the purposive sampling, notably that contacts in the North West and especially Scotland were very active in promoting the survey to students and trainees. The near absence of responses from Northern Ireland and variations between numbers in different regions are thought also to reflect this aspect of the sampling strategy more than any other factors. The distribution should provide opportunities to study responses from Scotland and the North West in some depth but limits the options in many other regions.
•
Responses from registered users were more evenly balanced with a quarter in London and the South East, which is the ‘home’ of the team managing the website.
•
For the purposes of the evaluation, at least some responses were received from right across the UK geographically and almost every UK medical school and postgraduate training deanery.
13
Evaluation of the NHS Medical Careers website
5.2 5.2.1
Awareness and use of the website amongst target audiences Awareness amongst potential users
A key issue for the evaluation was to obtain an indication of the level of awareness of the NHS Medical Careers website amongst its target user groups, especially medical students and postgraduate trainee doctors. Table 5.3 indicates that 49% of respondents within the ‘potential users’ sub-sample were aware of the website. In terms of site usage, 58% of potential users had never used the site, so 42% had used the website at least once. When analysed by training stage, it appeared that a higher proportion of Foundation Programme trainees were aware of the website, especially Foundation Year 2 (FY 2) doctors. However, 39% of Specialty trainees and 48% of medical students were aware of the site, which are also substantial proportions. In terms of usage, over half of Foundation Y2 trainee respondents had used the site, while the lowest proportion was of Specialty trainees, of whom it was just under a third. Table 5.3 Awareness of and usage of Medical Careers website amongst potential users
% Aware of site Used the site Count (N) Total 49 42 902 Medical students 48 43 455 Foundation Y1 trainees 57 44 75 Foundation Y2 trainees 67 59 109 Specialty trainees 39 31 238
Analysis of the level of awareness of the website by geographical region is provided in Table 5.4. Caution needs to be observed as there is much variation in the size of the response samples between regions used to derive the awareness percentages, some of which are very small. However, this suggests that awareness levels are around 45% for the regions with larger numbers of responses (which are probably the more reliable), quite similar to the overall level. Higher levels and a greater variation are seen in the regions with small numbers of responses; it is likely that sampling effects are masking any systematic variation due to genuine differences in awareness levels. What is perhaps most striking is the similarity of the levels of awareness in Scotland, the North West and the South East, which together represent about 70% of the responses in this sub-sample.
14
Evaluation of the NHS Medical Careers website
Table 5.4 Awareness and use of website by nation and region (for all ‘potential user’ respondents, N=963)
Nation England London South East South West East of England East Midlands West Midlands Yorks & Humber North West North East Scotland Wales Northern Ireland Region % aware of website 50 54 43 46 81 52 57 48 45 46 55 % who have used website 42 54 35 43 76 43 50 38 38 39 51 Number of responses N 576 35 95 28 37 23 40 288 29 307 45 -
Note: insufficient data for Northern Ireland and Yorks & Humber to derive percentage
The levels of usage are partly related to the awareness levels, and a roughly similar pattern emerges, with around 35-40% of the ‘potential users’ sub-sample having used the website at least once. Again the levels in the other regions, which have the smaller response numbers, are somewhat higher and also more variable. Although there are insufficient responses to undertake analysis by training stage within each region, it is evident from spot-checks of sub-groups of data that there may be variations. For example, proportionally fewer medical students in Dundee appear to be aware of the site than at Keele or Manchester, which in turn appear to be fewer than at Kings College London or Cambridge, and so on. However, there is insufficient data to conduct such analysis robustly by training stage even within the more deeply sampled regions. A significant number of medical student respondees indicated, using a free-text response, that they had not been aware of the site before and felt that it should have had greater promotion through medical schools. As examples, two such students put this rather
15
Evaluation of the NHS Medical Careers website
succinctly: ‘I'm confused why this hasn't been promoted at my medical school. I have very little knowledge of post-graduate training’ and ‘It should be promoted more within medical schools as a lot of students I know have not visited the website and don't realise how much useful information there is on there’. A proportion of those who had never used the site reported, however, that they had been aware of it when surveyed (7% overall of potential users, equating to 12% of those who had never used it). They were split roughly evenly between training stages (medical school, Foundation and Specialty training). These respondents were asked for what reason they had not used the site despite being aware of it. The majority indicated that that had not used it because they had not had a need to do so yet, although some postgraduate trainees reported that it was because they were unsure of its purpose. Very few (under 10%) said that it was because they would have no need for it at all or for any careers support.
5.2.2
Frequency of use of the website
A breakdown of the frequency of usage by those who had used the website is given in Table 5.5, combining responses from both the main sub-samples. This appears to demonstrate that, of those who had used the site, Foundation Y2 trainees and Specialty trainees were the most frequent users. Of those who had registered with the website, approximately 40% of all registered users had used the site more than three times, and a further 40% 2-3 times, which shows, not unexpectedly, that registered users generally use the site more frequently than users overall. Questions about the location from which access to the website was not especially revealing other than to confirm that all types of users made some access to the site from both home and their workplace or place of study. Access problems had only ever been encountered by fewer than 4% of users. Where specified, these seemed to be restricted mainly to problems with computers within NHS institutions which had very stringent limitations on access to internet sites, rather than any issues related to the provision of the website itself. Table 5.5 Frequency of use of website by training stage (respondents who had used the site)
% Once only 2-3 times > 3 times Count (N)
Total 28 41 31 730
Medical students 30 44 26 339 16
Foundation Y1 trainees 31 36 33 92
Foundation Y2 trainees 23 44 34 125
Specialty trainees 25 40 34 152
Evaluation of the NHS Medical Careers website
The attraction campaign for the surveys had not been designed to obtain quantitative information on the level of awareness of the site amongst the potential ‘professional’ target audiences, such as careers advisers/professionals and medical trainers/advisers. Responses obtained in these categories were largely from those who were registered users. Perhaps related to this sampling, most professionals who responded had used the site several times. The survey responses from professionals are reported in section 5.6, including their perceptions about the levels of awareness of the site amongst students and trainees. There was little consistency in their perceptions of the level of awareness of the site amongst medical students, but many of the professionals appeared to think that most trainees were not aware of the website. Interviews with advisers and trainers presented a range of perceptions on this, but mostly suggested that a significant and growing number of students and trainees were aware of the site, but with doubt about whether this was yet a majority in some institutions.
5.2.3
Summary and implications
The data obtained concerning awareness of the website is based upon a purposive sample specifically intended to ascertain responses from a diversity of potential users both geographically and by training stage. These appear to demonstrate that: • There is considerable awareness of the website amongst its main target audiences of medical students, Foundation trainees and Specialty trainees – of whom approaching half overall know of its existence, with the highest levels of awareness amongst Foundation trainees; • Around 40% overall of potential users sampled have actually used the website at least once, and nearly 60% of Foundation Year 2 trainees; • There is some awareness and use of the site by potential users across the nations of the UK and in all the regions of England. Sampling effects limit detailed analysis understanding by region but the levels of awareness in three diverse regions (Scotland, the North West and the South East) are all similar at around 45-48%. Levels of use of the site amongst potential users also appear to be roughly consistent at 35-40% in these different regions. • Web analytical statistics for recent months show an actual usage of 9000-10,000 unique visitors per month, of which about 90% are from within the UK. There has also been systematic growth in the number of registered users, which was c.3450 in early 2010 but was reported to be approaching 6000 in October 2010.
17
Evaluation of the NHS Medical Careers website
•
Awareness and use appears to be highest amongst Foundation Programme trainees, probably reflecting the site’s strong content targeted for this group and focus on provision of information about specialty choice. However, the levels of awareness and use amongst medical students and especially amongst Specialty trainees, who were considered to be relatively harder to reach, could be considered quite impressive given the relatively infancy of the website;
•
The apparently consistent awareness and usage figures across different regions (at least those with highest reliability sampling) also reflect success in promoting the website to a wide geographical audience;
•
Of those who have not used the site, the majority had not been aware of it and report that (now that they are aware of it) they expect to make use of it in future.
•
That a high proportion of Foundation and Specialty trainees especially who have used the site (c.75%) have returned to it several times strongly suggests that they value its content. Interviews with advisers corroborated this view, suggesting that trainees return to the site periodically at particular times of need, once they know of its existence and ‘bookmark’ it.
18
Evaluation of the NHS Medical Careers website
5.3 5.3.1
How and why users came to the website Reaching the site
The most common mechanism by which users had first come across the Medical Careers website was by searching the internet (37% overall and almost half of medical students). However, significant proportions said that they had followed a link from another website, seen reference to the site within careers information or had been recommended by a careers adviser or similar (all around 25-26% of users). This relative ordering accords well with website traffic statistics (Google Analytics) which recorded c.40% of users to have ‘arrived’ via search engines (chiefly Google itself), c.30% through links from other referring websites and c.30% coming ‘direct’ to the website, in recent months. When analysed by training stage, the survey data show that higher proportions of Foundation Programme trainees, than others, had been recommended the site by careers advisers (in fact as many as had found the site by online searching), while almost twice as many Specialty trainees as other groups had been recommended the site by their peers or friends (Figure 5.1). Other reasons cited by fewer than 10% of users are not shown.
Figure 5.1 Principal reasons/mechanisms that prompted users to access the website for the first time, by training stage (all site users, N=702; multiple responses were allowed)
web search website link reference in careers info career adviser recommendation trainer recommendation peer/friend recommendation promotional material
0 5 10 15 20 25 30 35 40 45 50
% Specialty trainees Medical students Foundation Y2 All respondents Foundation Y1
19
Evaluation of the NHS Medical Careers website
A number of the advisers and trainers interviewed were promoting the website strongly to their trainees, in the form of embedded references and hyperlinks within careers information materials, but also in some cases physically distributing leaflets. As one put it: ‘I’ve generated hundreds of documents all with embedded links to the site – they would all be out of date if it didn’t exist.’
5.3.2
Motivations to use the website
It was felt important to record the motivations of users in accessing the website, as well as some understanding of their context in terms of career and training decision-making. The most commonly cited primary motivation for using the website, for all those in training, was for greater awareness of career options, although the ranking of motivations varied according to training stage (Figure 5.2). For medical students, awareness of options was the most cited primary motivation (by 44%), while for Foundation Y1 trainees it was more for specific knowledge to inform impending decisions (42%), and for Foundation Y2 trainees even more specifically the need to select and apply for specialty options (57%).
Figure 5.2 Primary motivation to use the website (all respondents in education and training who had used the site, N=760)
greater awareness of career options
specific knowledge to inform decisions
select and apply for Specialty options
select and apply for Foundation positions
0
10
20
30 % Foundation Y2 All respondents
40
50 Foundation Y1
60
Specialty trainees Medical students
20
Evaluation of the NHS Medical Careers website
The in-depth views of advisers tended to support these quantitative findings, many believing that trainees, in particular, returned to the site mostly in times of need, when they had to focus on particular decisions or applications. This, they felt, was due to the other pressures on postgraduate doctors which meant that the time they could put into career-related research or investigation tended to be very limited, hence they would largely use the site on a ‘just in time’ basis rather than more systematically. This pattern was also suggested by a careers adviser in relation to medical students, whom he believed might use the site in exploratory fashion in their earliest years, but then return to it ‘in panic’ in years 5 and 6 as they neared the key times for next stage decisions.
5.3.3
Users’ engagement in career thinking
In order to understand the level of engagement of users in career thinking and learning, questions were asked about what other career-related activities they had participated in within the last 12 months. Widespread use of online careers information was evident, with around half of all users having accessed the BMJ Careers website, the MMC website and other career-related websites. Over half had engaged in career discussions with a third party, and about 10% had recently undertaken a formal careers education module within their studies. The proportions varied somewhat by training stage, with Foundation Y2 trainees consistently the most engaged (as shown in Table 5.6). An indication of the parties whom respondents had consulted about careers within the last year was also given, along with the nature of those discussions, shown in the lower half of Table 5.6. This also showed that informal discussions about careers were widespread and much more common than formal interviews or advice sessions with career professionals. The majority (80%) had talked about careers with peers and colleagues, and nearly half informally with medical faculty, and somewhat fewer with educational or clinical supervisors, which was far more than with careers advisers either informally (14%) or in an interview (12%). When analysed by training stage, Foundation Y2 doctors were again the most actively engaged in consulting others, while around two thirds of Specialty trainees had consulted their supervisors about career issues, but they were somewhat less engaged in informal discussions. Very low percentages reported that they had not thought about careers (with the exception of students, of whom it was 1 in 6) and almost none had not discussed careers with at least somebody.
21
Evaluation of the NHS Medical Careers website
Table 5.6 Other career-related activity undertaken by users (all respondents in training who had used the website)
% No career activity Careers module undertaken recently Used other career websites Used BMJ Careers website Used MMC website Career discussions undertaken Formal careers interview Informal with careers adviser Discussion with education supervisor Discussion with clinical supervisor Informal with medical faculty Informal with peers/colleagues No career conversations Count (N) All in training 10 9 47 52 50 56 Medical students 16 7 49 45 40 55 Foundation Y1 trainees 7 12 38 53 55 55 Foundation Y2 trainees 3 15 49 61 68 73 Specialty trainees 5 6 39 56 56 52
12 14 45 43 48 81 3 694
8 13 22 30 54 88 2 340
13 11 59 39 56 77 2 94
21 21 78 71 50 85 0 127
12 9 69 59 32 71 3 143
5.3.4 •
Implications Promotion of the website, and embedding reference to it within careers information and support, appears to have been relatively successful; many Foundation Year 2 trainees in particular learnt of the site through such reference or recommendation.
•
As for many websites, the survey suggests that most users (and especially students) came across the site through online means, either a search or through a direct link from another website, and this is backed up by web traffic statistics. This suggests that the site is reasonably visible amongst potential users.
•
The principal motivations to use the site are somewhat varied, but most seem to be seeking knowledge or awareness, potentially to support career-related decisions; even amongst postgraduate trainees many were seeking greater awareness as well as more specific information to support decisions and applications.
22
Evaluation of the NHS Medical Careers website
•
Questions about other engagement in career thinking and career-related learning activities suggest that informal discussions about careers are widespread amongst trainees – career learning is not restricted to formal sessions with careers professionals, and that there is widespread use of online careers information. Only 1 in 8 overall reported that they had taken part in a formal interview with a careers adviser or similar within the last 12 months, although this was about 20% of Foundation Y2 trainees. These indications of engagement provide some endorsement of the need for high-quality careers information on visible and reputable websites, to underpin informal discussions and self-guided learning.
23
Evaluation of the NHS Medical Careers website
5.4
Impressions of the website and its offer
Users expressed very positive views about the website, in terms of its design, appearance, organisation and overall content offer. The proportions of users indicating agreement with a selection of descriptive statements about the website are shown in Figure 5.3. Analysed by training stage, there was little difference between sub-groups of users, although some evidence that Foundation Y2 and Specialty trainees were slightly less positive about all aspects, and medical students more positive, than the overall user percentages. However, the differences were relatively slight, particularly if the proportions who did not remember are discounted from the analysis. What was also evident was that those who had used the website the most (the sub-group who used it more than 3 times) consistently expressed more positive views than those who had used it less, although here too the differences were not large.
Figure 5.3 Percentages of users expressing level of agreement with a series of descriptive statements about the website (all users, N=790)
Looks professional Brand inspires confidence Sufficient info range Enough info depth Up-to-date info Well-organised Easy to navigate
0 10 20 30 40 50 % 60 70 80 90 100
Strongly agree
Slightly agree
Slightly disagree
Strongly disagree
Don't remember
A note of caution should be introduced, however, based on interview dialogues with professionals, of whom several struggled initially to recall which website was under investigation (although remembered it in detail once prompted). This might suggest that it is not as distinctive or memorable as a brand or site, as some other medical websites. Respondents were invited to consider whether the website lacked any information or content that they had been expecting. Overall, 84% did not think that information or content was lacking; this figure varied very little by training stage or by how many times they had used the site. Views from professionals were every bit as positive (reported briefly in section 5.6).
24
Evaluation of the NHS Medical Careers website
The 16% who indicated that the site did not offer information that they had hoped for were given the opportunity to identify what they had sought, through an open-ended question. Those responses can be grouped and summarised, in decreasing order of frequency, as follows: • More detailed information on some specialties (some asked for sub-specialty level), including detail of specific training pathways within specialties – chiefly from postgraduate trainee doctors; • More descriptive information on specialties (e.g.“a day in the life”), to contextualise the existing formal information and standardised job descriptions; • • • Enhanced self-assessment tools specifically linked to specialty choice. Elements of workforce statistical information, especially competition ratios; Practical assistance and advice on interviews and applications;
However it should be stressed that these suggestions were from a minority of respondents, and many appeared to be very individualised requests; it must also be re-emphasised that the information currently offered on the website was satisfying 84% (the clear majority) of users. This was reinforced by views from the professionals within the survey, who also made few suggestions of additional content required, although a few intimated that more detail within the specialty pages could be even better. More in-depth dialogue with several advisers tended to support the possible areas indicated where content could beneficially be developed further, particularly in relation to specialty information. Several commented that additional content might usefully give greater insight as to what both training and eventual work in a particular specialty would be like, as they believed trainees struggled to envisage this and could not always find specialists whom to ask. Therefore more testimonials from trainees and specialists would be welcome, as well as more bespoke ‘what we would do when’ type of information in relation to detailed training pathways within particular specialties. The potential high value of ‘interpreted’ workforce statistics was also mentioned more than once, as the information available from the main statistical sources was considered of little direct use to trainees. These suggestions are considered further in the ‘Recommendations’ section of this report.
5.4.1 •
Observations The ‘first impressions’ recorded seem to present a very positive reaction to the presentation and design, ‘look-and-feel’ and organisation of the website, across the
25
Evaluation of the NHS Medical Careers website
different target user groups, of whom students in particular can be a relatively demanding audience. They are well summarised in the comment from a careers adviser: ‘I like the clean look of the site, uncluttered – easy to find my way around’. • Although based on a relatively small number of interviews, the lack of immediate recall of the website by interviewees may suggest that it has a less distinctive name or brand than, for example, the NHS Careers or BMJCareers websites (although there was no doubting the recollection of detail and content once they were prompted). • The substantial majority of current users reported high levels of satisfaction with the range, depth, type and currency of the information provided. Slightly lower levels of satisfaction with the nature of the content, from Foundation trainees relative to students, may simply be a reflection of their greater sophistication as an audience, and there was no systematic suggestion from professionals that the information provision was in any way lacking. • There was some consistency in recommendations, albeit from a minority, that additional detailed content could be beneficial in relation to specialties. This appeared to constitute some demand for more specific training pathway information within specialties, more testimonial or case study context, and potentially also additional self-assessment tools with which to support specialty selection. • The website management team is currently investigating how best to provide workforce statistics on the website; it appears from users that this sort of information (which they consistently referred to as competition or success ratios) would be beneficial if it could be provided in ‘interpreted’ fashion rather than as raw workforce statistical data.
26
Evaluation of the NHS Medical Careers website
5.5 5.5.1
Detailed usage and usefulness of content Usage of main content sections
Users were asked to indicate which of the main structural sections and features of the website they had accessed and the extent to which they had found each of them useful. Analysis of these responses shows that the ‘Specialty pages’, ‘Career options’ and ‘Career planning’ sections were the most widely used, each having been viewed by over 75% of users (Figure 5.4). ‘Tools and resources’ had been viewed by about two thirds of users, but other sections by fewer than half overall. However, some of those other sections were targeted towards particular user groups and had been strongly viewed by their respective target audiences; for example over two thirds of students had viewed the ‘Medical students’ section and over 60% of Foundation trainees had viewed the ‘Postgraduate doctors’ section. Some other sections targeted towards other specialist audiences, such as trainers, careers specialists, or those considering medicine, predictably had rather low overall viewing figures but this does not mean they were not well used or rated by the professionals. For example, all but three of the careers adviser respondents had viewed the ‘career specialists’ section and all who had viewed it reported that it was useful. The only generic sections which had been viewed by less than half of users were the ‘My profile’ section (60% not viewed) and ‘Links’ (52% not viewed).
Figure 5.4 Usage and usefulness figures for different sections of the website (respondents in education and training and had used the website, N=769; sections marked* rated by respective target audience groups only))
Specialty pages Career options Career planning Medical students* Postgraduate doctors* Tools & resources Links My profile
0 10 20 30 40 50 % 60 70 80 90 100
Viewed: very useful
Viewed: quite useful
Viewed: not useful
Intend to view
Not viewed
The level of perceived usefulness of content within these sections was also obtained through this question (also in Figure 5.4). For almost all the sections, about 30% of those who had viewed a section found its content very useful, about 60% quite useful and fewer than 10%
27
Evaluation of the NHS Medical Careers website
not useful, which appears to be strong endorsement for the content provided, even of the sections that were not viewed the most widely. The only section which differed somewhat was ‘My profile’ for which over 20% (of a smaller number of viewers) stated it had not been useful to them. Analysed by stage of training, there were some modest differences in the proportions of users reporting that viewed content was very useful, with the highest proportion amongst medical students and the lowest amongst Specialty trainees, but little difference between the proportions reporting it as quite useful or not useful. This could simply reflect that students are somewhat less discerning users/viewers than those with more experience (i.e. a higher proportion of them are more easily pleased). When analysed by frequency of use of the website, fairly consistent trends were observed. Perhaps unsurprisingly, frequent users had viewed more content than infrequent users. Of the frequent users: • • Higher proportions had viewed each section (in all cases), than of infrequent users; Higher proportions reported content in sections to be very useful, and somewhat higher proportions that it was quite useful, than of infrequent users; • Lower proportions had not viewed each content section than of infrequent users (generally about half as many); • The proportions who had viewed content but not found it useful were similarly low to those of infrequent users. Separately, asked to rate which single aspect of the website and its content was the most useful to them, using open-ended responses, respondents most frequently mentioned specialty information (in fact this was cited by almost two thirds who responded). The other aspects of content consistently mentioned by significant numbers of respondents were career planning and also self-assessment tools. The inverse question was also asked, i.e. which aspect of content was the least useful, although only about one in six made any response to this. Of those that did, the only aspects/sections mentioned with any consistency were the ‘My profile’ section (although only by 10 respondents) and that there was insufficient depth of information (by fewer than 10 respondents). In fact many respondents either stated that they felt all the content was useful or where something was not useful specifically noted that this was because it was not relevant to their particular training stage (but would be relevant to others). Hardly any respondents at all mentioned that either any video or interactive content was not useful.
28
Evaluation of the NHS Medical Careers website
5.5.2
Media-rich content and interactive functionality
The Medical Careers website management team has invested significantly in generating and hosting a number of videos to enhance the content provided, and also a series of interactive online tools to support self-awareness and career-related decision-making. It was felt important to evaluate the level of usage and perceived usefulness of these forms of ‘enhanced’ content, in order to gain some measure of the value of the investment made. Just under one in six users of the website (16%) claimed to have viewed at least some of the video content, although this varied by stage and frequency of use (Table 5.7). A somewhat higher proportion of Foundation trainees (about 20%) had viewed video content, but rather fewer medical students (12%) and very few Specialty trainees (6%). This almost certainly reflects the nature of the video content offered, of which much is concerned with specialties in order to support specialty decision-making. The variation with frequency of use was more striking still, with 33% of those who had used the site more than 3 times having viewed video, but only 5% of those who had visited the site only once; 21% of registered users had viewed video content. The relatively limited number of respondents who had viewed any video means that analysis by frequency of use within each training stage would not be robust statistically.
Table 5.6 Usage of video content and interactive online tools (respondents in education and training who had used the website, N=742))
% Viewed video content Used online interactive tool/s
All site users 16
Frequent users 33
Registered users 21
Medical students 12
Foundation Y1 trainees 20
Foundation Y2 trainees 18
Specialty trainees 6
27
47
38
29
28
24
17
The c.180 respondents who had accessed any video content were invited to indicate whether they had viewed particular video content from a list and their perception of how useful it had been. Although the numbers of respondents are rather small, certain specific usage and usefulness data are presented in Figure 5.5, for the series of videos about different specialties, consultants talking about their own specialties and trainees talking about their personal experiences, as well as two recently commissioned individual videos. This showed, broadly, that around half of those who had viewed any video had viewed the particular types/cases highlighted (i.e. roughly 100 viewers within the respondent sample). A series of
29
Evaluation of the NHS Medical Careers website
audiocasts appeared to be used considerably less than video content, while text-based case studies were accessed by slightly fewer than those who viewed videos. This indicates that the usefulness of the videos was highly rated, with very small proportions of viewers reporting that they had not found the video content useful. The usefulness of videos on specialties was particularly highly rated. Although the restricted numbers limit the analysis, it appeared that when analysed by stage of training, students seemed to rate video usefulness higher than other viewers. The proportions of registered users or frequent users who had viewed any particular video were about twice as high as the proportions of infrequent users. Figure 5.5 Usage and usefulness of specific media-rich content (respondents in education and training who had viewed video material, N=180)
Different specialties Consultants on specialties Trainee experiences Foundation advice General practice Audiocasts Text-based case studies
0 10 20 30 40 50 % 60 70 80 90 100
Viewed: very useful
Viewed: quite useful
Viewed: not useful
Intend to view
Not viewed
The interviews with advisers mostly revealed positive perceptions of the value of video content, as they considered it to be an attractive medium for students and trainees. It was also thought that video could offer more personal and contextualised information which trainees valued, rather than information as formal text. Although one considered it a poor substitute for the genuine experiential learning achieved working alongside specialists available during rotations – and the informal careers awareness picked up in doing so – most acknowledged that video was a pragmatic and valuable asset on the website. If anything the consensus was that there should be more not less video content, although with some videos of shorter length. A similar analysis was carried out for the interactive career tools offered on the website, use of which required registration on the site (other than for one specific tool). A somewhat higher
30
Evaluation of the NHS Medical Careers website
proportion of users had used at least one of the interactive online tools than had viewed video (27% overall). Somewhat more of the students (almost 30%) and Foundation Y1 trainees had used the tools than of Foundation Y2 and especially Specialty trainees (17%). The proportion of frequent users who had used the tools was much higher (47% of those who had used the website more than 3 times). The proportion of registered users was relatively high, unsurprisingly, at 38%. In parallel to the analysis of particular types of video, those who had used any of the career tools were asked which of the specific tools they had used and the degree to which they had found it useful (Figure 5.6). Each tool had been used by three-quarters or more of this group of users, and over 80% had used the ‘Careers values’ and ‘Personal influences’ tools. The perceived usefulness ratings of the different tools were all rather similar, with less than 15% of those who had any particular tool reporting it to be not useful. As with the video content, more of the frequent users or registered users were likely to have used a particular tool than of infrequent users. Interestingly, although the ‘Skills’ tool did not require registration, its use was not higher than of the others tools for which registration was necessary. The advisers and trainers interviewed were unequivocal about the interactive tools, believing them key to the unique offer of this website. While many would prefer that tools would be used in mediated fashion, i.e. with advisers, it was acknowledged that this was unrealistic on the scale demanded, so self-guided tools were now critical. Several commented that the Medical Careers website was uniquely valuable amongst online medical career support in making available (as one stated) ‘practical tools, in a no-nonsense way’. Figure 5.6 Usage and usefulness of specific interactive career tools (respondents in education and training who had accessed interactive tool content, N=230)
Personal influences Careers values Personality type Environmental factors Interests Skills
0 10 20 30 40 50 % 60 70 80 90 100
Viewed: very useful
Viewed: quite useful
Viewed: not useful
Intend to view
Not viewed
31
Evaluation of the NHS Medical Careers website
The level of use of a number of other specific and recent features of content was also probed in similar fashion. Specifically these were the Calendar, ‘Spotlight’ feature, a feature on ‘Working abroad’ and Links (to other websites). The level of usage of the first three of these was fairly uniform, with around 75% not having viewed these content features, while about 60% had not used the Links. In contrast with the video or interactive tools, however, the usage of these features seemed not to vary, or at least varied much less, with the type of user, frequency of usage or whether they were registered.
5.5.3 •
Summary of assessments of content The website’s content relating to specialty information, career options and career planning was the most widely used, by over 75% of users overall. This is confirmed by website use statistics (‘Google Analytics’) which record these sections as the top three in terms of sections of content viewed, with 1500-2000 page views per month of the opening page of each section.
•
Other sections of content targeted towards particular audiences were also widely viewed by those particular audiences. Advisers and trainers considered that a major focus of the site and its reputation was its information relating to specialties.
•
The usefulness or value of almost all the main content sections was highly rated; users struggled to think of content that was not of value or importance. Website analytical data record average site visit times of 4 to 5 minutes, which is much higher than for websites in general, and an average of 5-6 pages viewed on each occasion; this suggests some amount of focus on each page of content, indicating that there is significant information which is studied at some length (i.e. well used).
•
Although video content had been viewed by only 16% of respondents overall, this proportion was double (33%) for frequent site users. Assuming the sub-sample is representative of registrants, and with growth in registered users during 2010, this suggests that over 1000 trainees have viewed video content. Viewers rated video content on specialties and case studies as very useful and all other video content at least quite useful. Advisers endorsed the value of media-rich content such as video and podcasts and believed it was attractive to the target audience in training.
•
Usage data for the interactive online tools provided to support career decision-making might potentially have been different from video or other content as registration was required (other than for one particular tool). This was partly to enable users to save their results for future reference. However, all the tools had been used by over a quarter of all users, and no higher for the tool with unrestricted access, so registration
32
Evaluation of the NHS Medical Careers website
did not appear to be an issue. All the tools were highly rated as useful. Advisers considered these tools to be a key feature of the website and critically important to supplement guided careers support. • Careers and education professionals interviewed considered that the presence of video and interactive tools gave the Medical Careers website unique value and went a long way to differentiating it from other online sources. Taken together the evidence seems strong that the investment in media-rich and interactive content has been worthwhile and should be continued if possible.
33
Evaluation of the NHS Medical Careers website
5.6 5.6.1
Views from professionals Survey responses
Just over 80 medical education professionals provided responses to the survey, mostly prompted by the e-mails sent to registered users (the wider e-mail campaign was not overtly targeted towards professionals). Response options and certain bespoke questions had been provided for such respondents. These respondents were almost evenly split between trainers, careers advisers/professionals, and other medical education professionals, respectively, and the majority had used the website at least twice. Although the modest numbers limit the statistical value of their response frequencies, overall patterns in their responses to certain questions are reported here. The aims of the professionals in using the website appeared to be quite broad, with some respondents identifying with every response option offered and most with several. Careers advisers in particular seemed to use the site to keep up-to-date, while trainers were keen to use the tools and resources and the site generally to reinforce their support for career planning. Overall impressions of the content on the website were very positive indeed, especially from careers advisers. Most who gave an opinion considered the most useful content areas to be the specialty information and tools supporting the 4-stage career model. Careers advisers were very keen on the specialty pages but several also made reference to the ‘considering medicine’ section. One commented: ‘I directed several of my students to this [considering medicine] in the early stages of planning an application to Medical School. I think this was excellent as it had many detailed sections including sound advice for those who had been unsuccessful’. Although only about half of these professionals had viewed video content, and fewer than half had themselves used the interactive career tools, almost all who had done so rated their usefulness highly. One or two commented that video was a particularly good format in which to present specialty information. One of the specific questions posed to the professionals aimed to obtain their perceptions of the level of use and value of the site amongst its target audiences. While the number of respondents of each professional type was limited (20-30 in each), the following response patterns were clear: • All the careers advisers, and almost all the others, believed the site was valuable to medical students and thought that more medical students would use it if they knew about it; on the other hand they displayed considerable uncertainty when asked whether they thought most medical students were actually aware of it;
34
Evaluation of the NHS Medical Careers website
•
All the trainers and ‘other education professionals’ believed the site was valuable to postgraduate trainees, and almost all thought that more trainees would use the website if they were aware of it;
•
The trainers and ‘other education professionals’ were more certain about their perception of how widely the site was known by trainees – a clear majority felt that most trainees did not know about the site;
•
The careers advisers especially, and most of the other professionals, believed the website contained information that was not available elsewhere;
•
Almost every professional agreed with the statement ‘as a professional I find [the website] invaluable’ (just two trainers did not agree).
5.6.2
Insights from interviews
The careers advisers and education professionals interviewed, while admittedly a small sample of those working in the field, were also all very supportive of the development of the website and its contribution to their professional work. Without reservation all were routinely recommending it to their students or trainees, as a national source of information which they considered to be trusted, reliable, accurate and (critically for them) independent and nonjudgemental. They considered this to be necessary as much of the ‘local’ provision of information, with either deanery or Royal College focus, they felt contained an element of self-interest and/or marketing, which the Medical Careers website did not. At the same time they felt that the site offered a very useable combination of sufficient hard factual evidence with contextual information, yet without overwhelming potential users such as medical students. Inclusion of content relating to working abroad and issues like flexibility in training were considered to add particular value. Some used the website for their own purposes, either to research information or for familiarity so that they knew what they were recommending, but the majority seemed more to rely on it as one of their key sources to which they would refer trainees. Rather in parallel to their perceptions of trainees’ usage patterns, they too appeared to have ‘bookmarked’ its presence and appreciated its value as and when they had a specific requirement of it. Although most had not viewed all the video content or used all the interactive tools themselves, there was near unanimity in the perception that these aspects of content were both distinctive and valuable to those they were trying to support professionally. A number of advisers reported very distinct feedback that they had recently received from trainees whom they had referred to the site, although this was by nature anecdotal.
35
Evaluation of the NHS Medical Careers website
5.7
Views from non-users
Certain information was obtained through bespoke questions for respondents who had not used the website within the ‘potential users’ sub-sample. Reasons why some who were aware of the website but had not used it were reported in section 5.2.1; these seemed not to show that they had no need for the site or did not want to use it, but rather had not specifically had occasion or requirement to do so yet. Many of the ‘contextual’ questions in relation to current career thinking and other careerrelated activities put to users (which were summarised in section 5.3 and Table 5.6), were also put to the non-users. A selection of the results is given in Table 5.7 below. Certain differences between the activities and views of those who have used the Medical Careers website and those who have not are considered in the next section. Table 5.7 Career-related activities undertaken within the last year and state of career thinking of non-users (respondents who had never used the website, N=501)
% No career activity Careers module undertaken recently Used BMJ Careers website Used MMC website Interview with careers adviser Discussion with education supervisor Informal talk with medical faculty Good idea of which specialty to pursue Not thought much about career All in training 13 7 Medical students 18 8 Foundation Y1 trainees 7 14 Foundation Y2 trainees 2 11 Specialty trainees 8 6
55 23 13 46 53 59 15
46 8 11 15 61 49 20
42 24 21 55 64 61 12
56 39 25 82 57 72 5
66 38 11 79 42 74 7
Of those who had never heard of the website (which was 51% overall of the ‘potential user’ respondent sample), three quarters expected that they would subsequently use the site now that they were aware of it, and higher proportions of students and Foundation trainees (see Table 5.8). Very few respondents suggested that they did not have any need for such careers information provision.
36
Evaluation of the NHS Medical Careers website
Table 5.8 Expected future use of website, by training stage, for those who had not previously been aware of it
% Expect to use site in future Uncertain about future use Have no need to use site Count (N) Total 76 Medical students 85 Foundation Y1 trainees 81 Foundation Y2 trainees 89 Specialty trainees 60
17
13
19
11
23
7
2
0
0
17
464
236
32
36
145
These respondents were also asked to indicate which features of the website that they would expect to use or value (it is assumed they had not at this point looked at the website to check them; they were encouraged not to do so prior to completing the survey). This usefully gives some insight to the type of content which might be valuable to this group of potential users. Their expectations are summarised in Table 5.9, which shows that ‘detailed information on career options/pathways’ and ‘detailed information about specialties’ were sought by about three-quarters, but also ‘basics of medical career planning’ by over 60%. Over half expected to use the interactive career/decision tools and over a quarter video content about specialties. Analysed by stage of training, higher proportions of students expected to use or value all types of content listed, while Specialty trainees reported somewhat lower expectations, but almost half sought the interactive tools in particular (Table 5.9). Table 5.9 Features of the website that current non-users expected to use or value in future (percentage of respondents who had never used the website, N=501)
% Detailed info on career options/pathways Detailed information about specialties Basics of medical career planning Interactive personal career/decision tools Videos about specialties/consultants Video case studies of trainees None of this content Count (N) All in training 79 74 62 53 28 22 7 501 Medical students 90 84 84 57 37 28 1 257 37 Foundation Y1 trainees 81 83 74 52 21 17 0 42 Foundation Y2 trainees 80 76 53 42 22 9 2 45 Specialty trainees 67 58 31 48 20 20 16 157
Evaluation of the NHS Medical Careers website
5.8 5.8.1
Measures of impact Users’ perceptions of value and impact
Respondents who were users of the website were asked to indicate the impact of that use in a variety of ways. Although it may be considered a somewhat unreliable measure, due to imperfect recollections, respondents were asked to remember what they had done immediately after viewing the website on the last occasion. Roughly three-quarters claimed that they did remember their next immediate action; nearly half reported that they had thought about their specialty choices and applications, while over a fifth had talked to a peer or friend about what they had seen and almost 30% had visited another career-related website (multiple responses were allowed). A potentially more reliable measure is perceptions of actions that were prompted by their use of the website (but which were not necessarily immediate actions). Overall, almost half reported that they had reflected on the information they had obtained, while nearly a third had been prompted to access other career-related websites and 27% had returned to this website for more information (Figure 5.7). Figure 5.7 Activities prompted by use of the website, by training stage (respondents in education and training who had used the website, N=581)
reflected on information/learning visited other career website/s returned to this website for more made decisions about applications talked to an adviser/trainer about choices nothing prompted by website don't remember
0 10 20 30 40 50 60
% Specialty trainees Medical students Foundation Y2 All respondents Foundation Y1
Analysed by training stage, some differences were apparent, with more of the students reflecting on learning and obtaining more online information, but higher proportions of Foundation Y2 and Specialty trainees making decisions about applications and talking to
38
Evaluation of the NHS Medical Careers website
careers advisers or trainers about their choices. This seems to reflect well their stage of progression through training and their need for career decision-making. When analysed instead by frequency of use of the website, some markedly varying results were seen, as shown in Figure 5.8. This appears to show that more of the frequent users believed that the site had prompted them to action, than of infrequent users; notably higher proportions visited this or other websites, finalised their decisions and/or talked to an adviser about them. Figure 5.8 Activities prompted by using the website, by frequency of use (respondents in education and training who had used the website, N=581)
reflected on information/learning visited other career website/s returned to this website for more made decisions about applications talked to an adviser/trainer about choices nothing prompted by website don't remember
0 10 20 30 40 50 60
% frequent users registered users infrequent users All respondents
Although the limited sample size restricts its statistical validity, when analysis was made by stage of training and frequency of use of the website, it was evident that higher proportions of frequent user respondents at all stages were prompted to take actions than of infrequent users. Half of frequent student users sought more information from other websites, and 40% of Foundation trainees who were frequent users were prompted to talk to advisers about their choices, and 40% of Specialty trainees who were frequent users finalised their decisions with help from the site. Respondents were also asked about possible specific impacts on them of using the website personally, by expressing their level of agreement with a range of statements. Figure 5.9 charts the percentages in strong and slight agreement, and slight and strong disagreement, in relation to influence of using the website on specialty and Foundation decisions, whether
39
Evaluation of the NHS Medical Careers website
they had recommended the site to friends and colleagues and whether it had directly led to a conversation with a careers/training adviser. Over half agreed that the site had influenced their specialty choice (and almost 40% that they were considering a new specialty), and around 40% that it had influenced their choice of Foundation training and/or changed their ideas about Foundation training applications. Almost half agreed that using the site had led to a careers conversation with an adviser, while over 60% had already recommended the site to friends or colleagues. Expressed in the reverse direction, fewer than half (42%) felt that they had experienced no specific impact of using the site.
Figure 5.9 Perceived impact of use of the website, expressed as percentage agreement with statements (respondents in education and training, excluding those making ‘non-applicable’ response, N>150)
influenced specialty decision considering new specialty influenced Foundation choice changed ideas on Foundation led to adviser conversation recommended site to others no specified impact
0 10 20 30 40 50 % 60 70 80 90 100
Strongly agree
Slightly agree
Slightly disagree
Strongly disagree
Analysis of these responses with stage of training is given in Table 5.10, where the percentage expressing strong or slight agreement with each statement is shown for each sub-group. This shows almost two thirds of Foundation Y2 trainee respondents to have been influenced by the site in relation to their specialty decisions, and almost half of students to have been influenced in relation to their Foundation training choices. Equally, it had led to a careers conversation with an adviser for two thirds of Specialty and around half of Foundation trainees. For frequent users, the percentages demonstrating such impacts were higher in all cases than of the corresponding sample of overall users. Additional analysis by frequency of website use and stage of training showed higher percentages still for certain sub-groups. For example, over 70% of Foundation Y2 trainees
40
Evaluation of the NHS Medical Careers website
who were frequent users believed the site had influenced their specialty decision, and as many as half were considering a new/different specialty, although the limited sample size limits its statistical validity. Table 5.10 Percentage of different respondent groups expressing agreement with statements describing perceived impact of using the website (excluding those making ‘non-applicable’ response, N>150)
% Overall Frequent users 70 50 61 47 59 75 35 Medical students 62 48 47 42 36 62 48 Foundation Y1 trainees 52 40 38 28 49 57 37 Foundation Y2 trainees 64 23 37 38 52 64 37 Specialty trainees 54 36 68 65 41
Influenced specialty decision Considering new specialty Influenced Foundation choice Changed ideas on Foundation Led to adviser conversation Recommended site to others No specified impact
59 40 45 40 47 63 44
Within the survey, respondents were invited to make any additional comments about the website or their survey responses should they wish to do so, in the form of free-text responses. Over 100 respondents elected to do so, of which many expressed positive statements about the website in varying forms including gratitude for provision of the website, which they had found useful. A small number of these comments are highlighted here for illustrative purposes: ‘It’s helpful to have a reputable resource for careers information all in one place’ ‘I feel its a really good website but I had never heard of it before until stumbling across it on google. If it was highlighted to everyone at medical school i think it would be a great resource for all of us’ ‘I think it is a great website with very useful, relevant and up-to-date information which has been great for myself in deciding which areas of medicine I would be more suited to.’ ‘It is an invaluable tool for foundation doctors considering their specialist training career’ ‘Its an extremely valuable website i was delighted to find it.’ ‘It is a fantastic resource, and a real move forward from where we were previously (a lot of disparate information from a range of sources that was often difficult to find or navigate).’ ‘The site fills a gap in information provision as the NHS site tended to give broad view of Medical Careers. I think it has been a "godsend" to me as I seek to give up to date and relevant guidance.’
41
Evaluation of the NHS Medical Careers website
Perceptions of professionals about the general value of the website to its target audiences were reported in the previous section (5.6), as well as an indication of the reasons they used the site. The careers advisers, trainers and other education professionals who gave responses to the survey almost uniformly perceived that the site was of value to medical students and to postgraduate trainees, and all but two expressed agreement that it was ‘invaluable’ to them as a professional. These views were largely corroborated in the interviews with advisory and education professionals. Their widespread promotion of and references to the site in their work indicate clearly that they consider it to have value and potential impact for their trainees and students. The most direct question put to them in relation to impact was whether they would miss the website if it did not exist. All these interviewees responded to the effect that they would definitely miss the website should that ever be the case.
5.8.2
Derived impact measures
Theoretically at least, a more independent measure of the impact of using the website can be attempted by comparing levels of certain activities undertaken, and perceptions held, by those who have used the website with those for similar individuals who have not. While this may not in itself offer proof of causality – rather it may show association – it may be useful in corroborating perceptions reported by the users in section 5.7.1, of whom a good proportion claimed that using the website has had career-related impact for them personally. On this basis, a number of comparisons are made between the responses to certain questions by respondents who have not used the website and those who are frequent users (whom we assume to be those with most benefit or impact from the site). In relation to perceptions of their career awareness and decision-making, compared with those who had not used the website at all: • A higher proportion of medical students who were frequent users had a good idea of where to undertake Foundation training (77%, v. 47% of non-users) • Higher proportions at all stages were fairly sure they knew which specialty they would pursue (for example, 95% of Foundation Y2, v. 72% of non-users) • Lower proportions of students and Foundation Y1 trainees had not thought about training choices yet (14% v. 28% for students; 3% v. 15% for FY 1 trainees) • Lower proportions of students and Foundation Y1 trainees had not thought about their long-term career options (12% v. 20% for students; 6% v. 12% for FY 1 trainees).
42
Evaluation of the NHS Medical Careers website
For each of these perceptions, there appears to be correlation between those who had frequently used the website and a more ‘advanced’ position in terms of career thinking. That correlation or association is not evidence for causality, as it could be that those who used the website frequently did so because they were more ‘advanced’ in their career thinking, rather than the other way around. When similar analysis is made of a series of more definite actions or activities, some similar associations or correlations were observed, while they were absent in others. Thus, compared with those who had not used the website: • A much higher proportion of frequent users had visited the MMC specialties website (68%, v. 23% of non-users, overall), with a more pronounced difference for medical students (58% v. 8%) and Foundation Y1 trainees (80% v. 24%). Notably, recent web analytical data shows that only 4% of users came to the Medical Careers website direct from the MMC website. • A somewhat higher proportion of frequent users (other than Specialty trainees) had accessed the BMJ Careers website; • A higher proportion of Specialty trainees who were frequent users had undergone an interview with a careers adviser (20% v. 11%), although there were no significant differences for other stages; • A higher proportion of students who were frequent users had discussed their career with their educational supervisor (36% v. 15%), although there were no differences for other stages. • Somewhat more Foundation trainees who were frequent users had undertaken a careers module recently (c.20% v. c.12%), but no difference for students or Specialty trainees; • There was no discernible difference in the proportion of frequent users who had talked informally about careers with medical faculty. This seems to show that there are associations between being a frequent Medical Careers website user and undertaking some of these actions, but not others. An association in relation to website use is perhaps unsurprising, but the differential use of the MMC website is striking and unlikely only to be a result of some being more regular web users than others; rather it suggests that use of one site is prompting use of the other. The level of web traffic coming directly from the MMC site (indicated by web analytics) would not account for this level of association, which suggests that most is in the other direction, i.e. from NHS Medical Careers to the MMC site.
43
Evaluation of the NHS Medical Careers website
For the remaining activities, the differences mean some differential levels of certain careerrelated activities which are outside the regular curriculum (and small or no differences where all are expected to undertake the activity at that stage). This could explain the greater propensity for Specialty trainees to undertake an interview with a careers adviser, which few might be expected to do otherwise, and equally for students to discuss their career with their education or clinical supervisor. On the other hand, it appears that informal talks with medical faculty, and discussion of career with supervisors, are relatively commonplace for most postgraduate trainees irrespective of use of the website. As intimated earlier, these differences cannot in isolation be considered to be evidence of a positive impact of using the website on the level of career-related activity, or leading to more ‘advanced’ career thinking, but are more powerful as subsidiary evidence when combined with the views of personal benefit expressed by the users themselves.
5.8.3 •
Summary of evidence for impact There seems to be unequivocal quantitative evidence, from trainees and students, that the majority within these groups perceive the Medical Careers website to be valuable to them.
•
Although the number of professionals responding to the survey was modest (c.80), they presented very strong views both of the value of the site to themselves as professionals (all but two said it was ‘invaluable’) but also their perceptions of its benefit to those in education and training. This was confirmed in the interviews.
•
Professionals reported distinct individual cases where trainees had used the website and had fed back that they had found it useful in coming to decisions and making applications (which are a different form of positive perception). These are reinforced by comments made by individuals within the survey which attest to very positive views as to the value of the website (with almost no views expressed that it is not useful).
•
There is also evidence of an association between heightened levels of certain activities indicative of greater engagement in career thinking and greater use of the Medical Careers website, some of which are more distinct for certain participant groups. This does not independently prove impact of the website, as causality cannot be proven (it could be that use of the website results from those very activities). However, taken with the very positive perceptions, it can be considered as reinforcing evidence for the impact of the website on its target audiences.
44
Evaluation of the NHS Medical Careers website
6.
6.1
SUMMARY OF FINDINGS AND RECOMMENDATIONS
Overall findings
The research study sought to obtain substantial quantitative evidence from users and potential users in relation to awareness, utilisation and possible impact of the NHS Medical Careers website amongst its target audiences. Online survey responses from potential users, chiefly medical students and postgraduate trainees, were used to measure their awareness of the website as a national careers information resource, and whether they had used it. Responses from those who had used the website were combined with responses from registered users in order to investigate use of the website, and in particular its value and potential impact. This included motivations for use, perceptions of the content and its presentation, and indications of which content had been used and how valuable it was, together with measures of perceived impact. The information was deepened through selected in-depth interviews and analysis of qualitative information supplied through the surveys. The research strategy aimed to provide a ‘purposive’ sample of target audiences in different regions of the UK, to assess the reach of the website, together with a more representative sample of website users for detailed investigation of use and content. Significant numbers of responses were achieved from all the website’s target audiences, in terms of both training stage and geographical location. A rather uneven geographical distribution of responses from potential users was achieved, but included respondents from almost every UK medical school and every postgraduate training deanery.
6.1.1
Levels of awareness and use
The survey revealed that there is considerable awareness of the website amongst its main target audiences – approaching half of medical students and those in Foundation and Specialty Training Programmes know of its existence, with the highest levels of awareness (over 50%) amongst Foundation trainees. Around 40% overall of these ‘potential users’ had actually used the website at least once, and nearly 60% of Foundation Year 2 trainees. There is awareness and use of the site by its target audience across all four nations of the UK and within all the regions of England. Uneven results of sampling did not allow analysis of variations in awareness or use by individual region, but suggested relatively even levels of awareness across a diversity of regions (notably Scotland, the North West and the South
45
Evaluation of the NHS Medical Careers website
East all similar at around 45-48% awareness). Levels of use of the site also appeared to be roughly consistent at 35-40% in these diverse regions. Awareness and use are highest amongst Foundation Programme trainees, but the levels of awareness and use amongst medical students and especially Specialty trainees are impressive given the relatively infancy of the website. Most potential users who had not been aware of the website seemed enthusiastic to use it in future, and many students in particular reported surprise that it had not been promoted to them by their medical schools. Roughly 80% of site users have accessed the site more than once, and half of these more than three times, and more still of postgraduate trainees, which indicates their perception that its content is valuable – so they are likely to return to it in times of need. Although most users (especially students) reached the site via online search or a link from another website, substantial numbers were referred to or recommended it by careers advisers and their material, suggesting that the site is well embedded in the minds of many medical careers and training professionals. Principal motivations to use the site are varied but principally knowledge or awareness either generally or to support specific career-related decisions. Only 12% of users had participated in a formal interview with a careers adviser or similar in the last year (20% of Foundation Y2 trainees), which endorses the need for high-quality online careers information to underpin more widespread informal discussions and self-guided learning.
6.1.2
Detailed utilisation and usefulness of content
Users were almost unanimously impressed with the design, presentation and organisation of the Medical Careers website, across the different target users. They were highly satisfied with the range, depth and currency of the content, almost all of which they found useful. On the other hand some struggled immediately to recall which medical careers this was, amongst many offerings, which suggests that it could benefit from a stronger identity. The website’s content relating to specialty information, career options and career planning was most widely used, by over 75% of users. Sections of content targeted towards particular audiences were widely viewed by those audiences. Advisers and trainers confirmed that specialty information was a major asset and a main focus of the site and its reputation. However, the usefulness or value of almost all main content sections was highly rated; in fact users struggled to think of content that was not of value.
46
Evaluation of the NHS Medical Careers website
Although video content had been viewed by only 16% of users, this was 33% for frequent users. Users rated video content on specialties and case studies as very useful. Advisers endorsed the value of media-rich content such as video and podcasts for these audiences. Interactive online tools to support career decision-making had been used by over a quarter of users, and were rated as very useful by most. Advisers considered these tools to be a key feature of the site and critically important to supplement guided careers support. Several careers/education professionals interviewed considered the video content and interactive tools made the NHS Medical Careers website unique, differentiating it from other online sources. This suggests that the investment in media-rich and interactive content has been worthwhile and should be continued if possible. There was some consistency in recommendations that additional detailed content could be beneficial in relation to specialties, such as more ‘trainee-focused’ training pathway information specific to specialties to enable them to understand what they would do, as well as more testimonials and case studies, and potentially additional self-assessment tools to support specific specialty choices. ‘Interpreted’ workforce statistics (referred to as competition or success ratios) were also identified as an area for content development.
6.1.3
Value and impact of the website
There was unequivocal quantitative evidence, from trainees and students, that the majority perceive the NHS Medical Careers website to be valuable to them. Medical careers/education professionals presented strongly positive views about the site’s value. All but two said it was ‘invaluable to them professionally’ and all believed it was valuable to trainees and students, and that more would use the website if they knew about it. They believed the website contained information that was not available elsewhere, as well as providing an invaluable first port of call, from which there would be links to deeper information if needed. Their perception was that the site was unique in combining detailed information presented in a non-judgemental and user-friendly manner, together with high value assets such as video and interactive tools. There is also evidence of an association between high levels of certain career-related activities and attitudes, indicative of a greater engagement in career thinking, and more frequent use of the Medical Careers website, although not all distinct amongst all audiences. While this suggests association rather than necessarily causality or impact of using the website, it seems to be evidence that reinforces the powerful perceptions from users and professionals that the website is having positive impact on its target audiences.
47
Evaluation of the NHS Medical Careers website
6.2 1.
Recommendations While this study cannot make any judgement as to whether the Department of Health’s investment in developing the NHS Medical Careers website represents good value for money, there seems to be clear evidence that it is valued by its target audiences and would be missed were it not to continue as a discrete website. On that basis, the first and perhaps most fundamental recommendation is that the site should continue to exist and be supported. (Sections 5.8.1 and 5.6)
2.
Based on the perceived value of the site to its medical student and postgraduate trainee audiences, it is recommended that the site continue to be promoted to those audiences so that the levels of awareness and use rise further, from what are already quite impressive levels given the limited period the site has existed. Promotion via medical schools would seem to be of particular merit, if achievable, so that new users can obtain sustained benefit from the site throughout their medical training. (Sections 5.2.1 and 5.8.1)
3.
Efforts should continue to promote the website through careers and education professionals, so that it becomes even more embedded within the careers support landscape, and to maintain its current prominence in relevant online search results. If necessary the latter could be bolstered periodically through attention to search engine optimisation. These two mechanisms should have impact on the two main routes by which current users reach the website, although promotion using multiple channels should be continued, making use of all cost-effective opportunities available. (Sections 5..3.1 and 5.6)
4.
Perceptions of the quality and nature of existing content are very positive, reflecting the firm content quality control procedure that is in place. Attention to sustaining this should be maintained, to ensure continued quality, currency and accuracy of specialty and other information. Maintaining the perceived ‘impartiality’ of the site seems particularly important, including features of content (such as working abroad, and training flexibility) which appeal to a variety of potential users. (Sections 5.5.1 and 5.6.2)
5.
Further development of content in relation to specialties would be welcomed. In particular this might take the form of detailed but ‘human’ information, such as more specific content so that users can envisage what the training pathway would be like for them, as well as more to help them understand what working in a particular specialty
48
Evaluation of the NHS Medical Careers website
‘would really be like’. This might incorporate some further video content, and possibly additional tools to support decision-making in relation to particular specialties. (Sections 5.4 and 5.6.2) 6. Provision of more workforce statistics is recommended if this can be presented in ‘interpreted’ form, as Collins (2010) also suggests. Much current statistical data available on other sites is hard for users to understand or interpret, and it is competition or success ratios, and future trends, that are the most important aspects to them. (Section 5.4) 7. In order to understand better the potential roles of different national medical careerrelated websites, and how they inter-relate, it could be useful to investigate in more depth the pattern of traffic (users) between them. This would be relatively simple but would require dialogue with respective providers/managers, but could add value to existing statistics which tend only to report single directions of web traffic (i.e. referrals to their particular site). (Sections 5.3.1 and 5.3.3) 8. Consideration might be given to the identity and name of the Medical Careers website, in order to give it a stronger identity and increase ‘recall’ amongst users. (Section 5.4)
49
Evaluation of the NHS Medical Careers website
7.
REFERENCES
Department of Health. (2004). Modernising Medical Careers – The Next Steps. The future shape of foundation, specialist and general practitioner training programmes. Department of Health. (2003). Modernising medical careers: The response of the four UK Health Ministers to the consultation on ‘Unfinished Business: Proposals for reform of the senior house officer grade’. British Medical Association 2003. Sign-posting medical careers for doctors. Accessed on www.bma.org.uk British Medical Association cohort studies. Accessed on www.bma.org.uk Collins, J.P. (2010) Foundation for Excellence: An Evaluation of the Foundation Programme, Medical Education England Darzi, Lord A. (2008) High quality care for all: Next Stage Review final report, Department of Health Hallam, R (2009) Lessons learned: medical careers website. London KSS Deanery. Jackson, C.J, Ball, J.E., Hirsh, W. and Kidd, J.M. (2003). Informing choices: the need for career advice in medical training, National Institute for Careers Education and Counselling (NICEC) Modernising Medical Careers Working Group for Career Management (2005) Career management: an approach for medical schools, deaneries, Royal Colleges and Trusts. Patel, S. G., Ahmed, R., Rosenbaum, B. P. and Rodgers, S. M. (2008). ‘Career guidance and the web: bridging the gap between the AAMC careers in medicine website and local career guidance programs’, Teaching and Learning in Medicine, 20 (3): 230-234 Tooke, Professor Sir J. (2007) Aspiring to excellence: final report of the independent inquiry into Modernising Medical Careers. MMC Inquiry, London
50