Blog posts

Fine Tuning Django User Permissions

Fine Tuning Django User Permissions

Dr Dan Davies from the Imperial RSE team has written a how-to guide based on his experiences with the Django web framework for python. Read the full blog post here.

The RSE team is involved in an increasing number of software projects requiring a front-end web app. The main advantage to having a web app element for your research software is that users can interact with it via a web browser, without having to install anything to their local machine. There are of course downsides, including the need to deploy, host and maintain software somewhere suitable. However, there is a wide range of popular frameworks to make the whole process a lot smoother.

User permissions are an important consideration for any web app. This is not necessarily just to do with overall security, but how you might want different types of users – with different roles – to interact with your software. For example, it is common to require admin users to be able to perform a wide variety of actions, while the majority of users should only be able to perform a small subset of actions. The degree of complexity required will depend on the overall aim.

We frequently use the Django web framework, which facilitates the creation of web apps solely in Python. This blog post covers aspects of user management and permissions within Django, which Dan has learned about and implemented while working on a web-based database to store and visualise sets of experimental data. It covers some basics such as how to assign permissions to user and groups of users, as well as more advanced topics such as setting up automatic permissions when specific objects are created. We hope it will be useful to the wider RSE community and beyond!

Simple permission assignment in Django
Fig. 1: Simple permission assignment in Django
Automatic permission assignment for specific objects in Django.
Fig. 2: Automatic permission assignment for specific objects in Django.

Building Research Software Communities

Building Research Software Communities: Running a workshop on community building and sustainability for the research software community

Michelle Barker, Jeremy Cohen, Daniel Nüst, Toby Hodges, Serah Njambi Rono, Lou Woodley

On Wednesday 17th March 2021, around 50 individuals from a wide range of different countries and time zones came together for the first of two 2-hour sessions that formed our “Building Research Software Communities: How to increase engagement in your community” workshop.

Run as part of the SORSE Series of Online Research Software Events, this workshop brought together an organising team consisting of 3 members of the international research software community and a group of speakers including experts in community engagement and sustainability. In this blog post we provide an overview of the workshop and some of the key messages and outcomes.

Why run a communities workshop?

The workshop’s three organisers – Michelle Barker, Jeremy Cohen and Daniel Nüst – between them have experience of starting and running, or participating in a range of research software communities at local, regional, national and international levels. Observing that many research software communities face similar challenges when getting started or trying to sustain activities, the workshop was set up with the aim of helping to address these issues.

Scientific or research communities are often set up by enthusiastic individuals who are keen to help their peers, raise the profile of their field and provide opportunities for training, knowledge exchange and networking. After what is often an extremely promising start with many people engaging and lots of attendees at initial events, it’s quite common for a community to lose momentum and for numbers to reduce to a small but committed group of people. Community organisers may begin to wonder where they went wrong, what they could have done differently and why people are not participating in the same numbers. Many people in the research software community are now involved in developing or helping to run communities (such as national Research Software Engineering (RSE) organisations) or want to initiate grassroots activities, but they are often without the experience or training to do so. The aim of this workshop was to try and offer some ideas, guidance and training from a group of scientific community engagement and sustainability experts to help begin to address this.

The workshop

The workshop included a combination of lightning talks and longer sessions run by leading experts in community engagement and sustainability from the Center for Scientific Collaboration and Community Engagement (CSCCE) and The Carpentries. You can find the full agenda on the workshop webpage. It was attended by participants with varying degrees of responsibility for, or interest in, managing research software communities.

Starting with a group of 4 lightning talks to set the scene, we heard from both current and former community managers representing communities at different stages of development. This provided a great opportunity to hear about some challenges faced but also success stories. Following the introductory lightning talks we had our first collaborative session of the workshop with Daniel Nüst running a short group feedback session.

Your three biggest community challenges

In the feedback session, participants were split into breakout groups, each with their own collaborative document, and invited to discuss and note down the three biggest challenges they’ve experienced/observed as a community manager or member. The results from this session helped to guide the discussion during subsequent sessions. The session provided a wide range of interesting and helpful responses which were summarised into five core areas:

  • Engagement – Keeping community members interested and engaged; managing challenges around limited time availability and workload issues
  • Incentives – Different environments (e.g., online / in-person) provide different motivation or incentives to participate in a community; what benefits/opportunities/activities incentivise participation?
  • Expectations – Be realistic about what a community can offer or what to expect from a community
  • Communication – Keeping community members informed; reaching out to potential new members; highlighting community aims and activities, etc.
  • Participation – Will people participate? How long will they participate for? How do you maintain participation?

Describing member engagement with the CSCCE’s Community Participation Model

After a chance for the workshop participants to discuss their community challenges, Lou Woodley, Director of the CSCCE, ran a session looking at “Describing member engagement with CSCCE’s Community Participation Model”. One of the key elements of this session was the presentation of the CSCCE’s Community Participation Model which defines four modes of member engagement that can take place within a community and one meta-mode – the champion mode discussed on day two. Community participants generally begin to engage with the community in the consume mode, taking in the materials that are made available through, for example presentations at events and online content such as newsletters. Levels of engagement can build through contribution and scaffolded collaboration to the highest level of engagement – co-creation – where participants work within the context of the existing bounds of community activity to create something new.

Download a guidebook describing the model in full here.

Community Champions

After a quick recap of the previous day’s material, the workshop slot on day 2 kicked off with a session from the CSCCE on Community Champions. The champion mode is the fifth mode in CSCCE’s Community Participation Model and highlights member engagement by emergent leaders within a community, who take on roles to maintain, grow and evolve the community’s activities. This might look like co-chairing working groups, serving on a code of conduct committee or spreading the word about the community to recruit new members. Lou Woodley highlighted the principles behind developing community champions and the important role that they can play in supporting community sustainability and ongoing engagement – something a community manager is unlikely to be able to do alone.

Community Sustainability

The final session of the workshop was a collaborative session on community sustainability run by Toby Hodges and Serah Njambi Rono from The Carpentries. Toby and Serah highlighted the challenges in ensuring community sustainability and presented various ideas to help address them. Using a collaborative document, a number of thoughts and comments were gathered from workshop participants in response to some important questions around the topic of sustainability. You can read more details about this workshop session in Toby and Serah’s “Pondering on the Question of Community Sustainability” post on The Carpentries blog and see a video of the session on the workshop’s SORSE event page.

Useful Links and Further information

This workshop was run as part of the SORSE “Series of Online Research Software Events. The SORSE series has now finished but you can take a look back at other SORSE events, many of which cover related topics, and see videos from many of the events via the SORSE Programme page.

Videos from parts of this communities workshop are available on the workshop’s SORSE event page and further details including the full agenda and session descriptions are available on the workshop website.

Why not join the CSCCE’s Community of Practice on Slack? It’s a great place to gain new knowledge about community development, engagement and sustainability and to share your experiences and questions.

 

The content of this blog post is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) licence. The post has also been published on the de-RSE and CSCCE blogs.

Research Software Directories

This is a summary of a SORSE discussion session, presented by:

  • Mark Woodbridge, Imperial College London
  • Vanessa Sochat, Stanford University
  • Jurriaan Spaaks, Netherlands eScience Center

And featuring contributions from:

  • Malin Sandström, INCF
  • Alexander Struck, Humboldt University of Berlin

Introduction

The discussion session “Research Software Directories: What, Why, and How?” was held on September 16 during SORSE, an International Series of Online Research Software Events. As presenters, we each shared efforts to develop and maintain software directories: catalogues to showcase the software outputs of an institution or community. The directories presented were:

Each of the above offered several advantages and disadvantages, or were scoped for particular use cases. For example, research-software.nl provides a robust application for serving detailed metrics and metadata for software, however it requires more manual entry. The Research Software Encyclopedia is automated and does not require hosting, but it lacks the same level of metadata. The Imperial College London and GitHub Search research software directories offer much quicker to deploy solutions, but might be too simple for some use cases. The directories are discussed in detail in the following sections. In addition to this set, we suggest the reader take a look at the Awesome Registries list to find additional examples.

How many participants use software directories?

We were quite surprised at the results of asking attendees the extent to which they have contributed or used software directories. For a total of 27 participants, 43% have used a directory for a relevant project, 27% have submitted software to a directory, and 58% indicated neither of the above.

Presentations

The Research Software Directory by Netherlands eScience Center

Jurriaan’s presentation started off by explaining why the Netherlands eScience Center had a need for what eventually became the Research Software Directory. Primary reasons were that as the Netherlands eScience Center grew beyond say, 20 or so engineers, tracking what software was available in-house really became too difficult a problem to do ad-hoc, despite the fact that all of their repositories are publicly accessible on GitHub. Secondly, the eScience Center strives to be as open as possible, and they thought it was important to be able to show the outside world where the taxpayer’s money had gone. Lastly, the eScience Center has a continuous need to keep track of various metrics, both for reporting to their funders (SURF and NWO), but also for helping management make informed business decisions.

Jurriaan then demonstrated the eScience Center’s instance of the Research Software Directory. While walking the viewers through the design, he explained how the product pages’ design was helping site visitors on their way towards adoption of the software presented on the product page.

When designing the Research Software Directory, specific attention was paid to how an instance is filled with data, how this data is curated, and how to do this in a way that can be sustained over time. To this end, the Research Software Directory harvests much of its information automatically, for example using APIs to GitHub (code development platform), Zenodo (archiving service), and Zotero (reference manager). This setup allows engineers employed by the Netherlands eScience Center to stay mostly in their comfort zone (i.e. GitHub). They just need to make sure to follow best practices such as having publicly accessible repositories, making releases on Zenodo using the automated integration, and including software citation metadata (CFF) in their repositories. Given that they already do much of that anyway, making an entry in the Research Software Directory can be achieved in a few clicks via the Admin interface that the Research Software Directory provides.

The Research Software Directory has proven to be a great resource for managing the organization, for providing funders with relevant metrics, and for increasing the visibility of tools. Despite these upsides, of course there are some downsides as well, for example it has proven difficult to carve out enough time to curate prose on the product pages, leading to text snippets that are sometimes too difficult to read for visitors not yet familiar with the software that the product page presents. A second problem is maintenance of the Research Software Directory software itself: the software stack includes more than 40 techniques, methods, and tools, in various languages and using a variety of frameworks. It has proven difficult to find developers that are familiar enough with all of these to be effective at maintaining the site. While this has not led to any significant downtime in the 3 years research-software.nl has been running, eScience Center intends to start reducing the software stack in the very near future. Furthermore, they are investigating whether it’s feasible to provide Research Software Directories as a service.

The Research Software Directory by Imperial College London

Mark Woodbridge demonstrated Imperial College’s Research Software Directory, explaining how it was developed to present a manually curated list of GitHub and GitLab repositories – motivated by a desire to rapidly catalogue and demonstrate the breadth of software developed at Imperial. It is also intended to encourage collaboration by assisting researchers to identify existing expertise and projects at Imperial.

The chosen approach has resulted in a system which is easy to maintain – both in operational complexity and in adding entries to the directory (even if the latter does depend on some familiarity with git and GitHub i.e. making a commit and pull request). This simplicity comes at a price: it depends on Algolia (a freemium service), has limited features, and is not easy to customise. It also relies on manual curation and repository metadata: due to limited bandwidth and lack of incentives, developers rarely submit or annotate software themselves. Finally, it lacks the polish and level of detail that you might expect of a public-facing showcase.

The system has however achieved its aims in effectively showcasing research software and developers at Imperial, and has provided a set of metadata enabling the identification of preferred languages to fast-growing fields of research. A suite of standalone utility scripts ensures that the contact details and project web pages remain up-to-date, and that new repositories by known developers are added to the directory in a timely manner.

The Research Software Encyclopedia

The Research Software Encyclopedia (RSEPedia) is a community-driven, open source directory that provides a means to communicate about software. It consists of three components – a set of criteria and taxonomy items used to describe or otherwise communicate about software categorization preferences, a database, and a command line client to interact with those components. The criteria and taxonomy items are maintained in their own GitHub repository, https://github.com/rseng/rseng, and render to an interface to allow for exploration and visualization. Importantly, the site for these items hosts a weekly software showcase, allowing the community to learn more about open source libraries that might be useful for their work. The terms are also served programmatically to a RESTful application programming interface (API) that makes them readily available for the RSEPedia software, which is also provided on GitHub (https://github.com/rseng/rse). Using the software, an individual or institution is empowered to easily generate a database and interface for a set of software they care about. They can inspect, add, search, or otherwise interact with metadata. While relational databases can be created, the community maintained database is a flat file database hosted on GitHub (https://rseng.github.io/software) that allows an interested contributor to browse, and annotate software with criteria and taxonomy items in an online interface. Annotation only takes a few clicks, and the process to make changes and update the database is fully automated via GitHub actions. Annotation in bulk is also easy to do locally after cloning the software repository, starting the annotation interface, and opening a pull request with changes. Importantly, although annotation can help to share ideas about software, it is not required to make the RSEPedia useful. By way of being able to communicate about software via asking questions, and by way of the software showcase, the RSEPedia can be successful for your needs if you just need a way to describe what you are looking for (e.g., for a grant or journal) or just want to share your set of software to be easily searchable.

GitHub Search is a derivation of the Research Software Directory by Imperial College London, but it removes the Algolia dependency, and derives software repositories directly from the GitHub API list of repositories for an organization directly on GitHub pages. This means that deployment is easy, coming down to simply creating the repository with a GitHub action to build it at some frequency to update the pages.

Discussions

After the presentations, attendees were divided over three groups for a 20-minute discussion session. All three groups saw lively discussions and discussed a plethora of relevant subjects, a selection of which is included below.

How do software directories interact with high performance computing (HPC)?

With several attendees that work as administrators for HPC, the question quickly came up about the relationship between software directories and HPC centers. Indeed, these centers typically maintain a large catalog of software for a user base, and it could be beneficial to link this software catalog or strategy to maintain it with a software directory. For example, if you are familiar with spack or easybuild you could imagine having integration to use a software directory to look up metadata, or generate user-friendly documentation pages. The pages might have install instructions, examples, and optimization hints for different architectures.

Guix-HPC is a package manager for a variety of software that is developed to allow reproducible HPC environments. It may interact with existing instances of Research Software Directories.

Curation policies

The main concern related to the “curation” of software directories were criteria for inclusion. A lively discussion related to the definition of “research software”, particularly in relation to scale and licensing. In the broadest sense there was agreement in principle that it could refer to any tool or library used to produce scientific results.

In terms of scale, attendees working in life sciences research emphasized that research software in their context could be a standalone script, and software directories should therefore “scale-down” appropriately.  Scripts of this type may be less substantial but their quality could well be assessed similarly to more prototypical projects in terms of documentation, design for re-use and version control.

Licensing was a more challenging topic – an argument was made for directories enabling users to find any tool that might accelerate research, including commercial software  – as long as an appropriate licence was available.

In broader terms, there was consensus that curators should avoid making assumptions about software applicability and relevance, even if they do have domain knowledge. More important than strict policies is effective annotation and filters so that users can apply their own criteria when searching for relevant software.

Searching for software

Searching for software presents its own challenges as an RSD only presents local results and many other platforms would need to be consulted for an exhaustive overview of relevant packages. Here, some registry lists prove to be helpful, for example Awesome Research Software Registries.

The purpose and minimum features of Research Software Directories

Participants identified discoverability as a major issue in relation to research software, particularly for domain specialists (i.e. end-users). This led to the following features being considered of primary importance:

  • Metadata clearly explaining the purpose and value of individual software tools in non-technical terms. The community is currently working on metadata standards like CFF or CodeMeta.
  • Contact details for the authors of the software in case further advice or support is required
  • Installation and getting started instructions
  • Guidance on how to cite the software
  • Licensing terms. This was discussed not only in relation to terms of use but also, for non-free software, ensuring cost-efficiencies by avoiding unilateral purchasing decisions and promoting the use or procurement of shared/group licences.

Many other features may benefit researchers, for example, linking from an RSD entry to its accompanying paper and data, as suggested in “Generalist Repository Comparison Chart” or listing received software citations, as implemented in swMATH.

Organization-based registry vs community-based registry

Some registries out there are scoped to serve an organization, whereas other registries like ascl.net or bio.tools aim to serve an entire research community. An advantage of the latter is increased traffic to the registry, and real benefits for users to browse the registry to see if somebody else in the community already created a solution. However, because the social structure across the community is quite loose, it will be more difficult to keep people involved, to discover new tools that could be added to the registry, and to make sure that the language used on the registry’s pages is understandable by everyone in the community. Furthermore, governance of the instance will be more difficult. For example, within the community there may exist different opinions on what metadata should be kept, and weighing these opinions will be more difficult in a larger community than a small one.

In contrast, organizational registries are more easy to run and govern — discovering tools that could be added is (or used to be) a matter of hanging out at the coffee machine and asking your colleague what they are working on right now. Helping your colleague enter their data, and making sure they do it correctly, is easier as well, and some good old-fashioned peer pressure can be applied if needed. Funding policies currently do not mandate the publication of research software, as Horizon 2020 required for research data (if possible).

Further resources

Recommendations and Next Steps

By discussing topics of curation, federation, technology and sustainability of research software directories with a wider audience, this discussion section hoped to not only promote the benefits of such directories and encourage their deployment, but also to identify issues and gather ideas to address them. From discussion above, it’s clear that there are interesting projects and updates to existing directories that might be pursued.

Remote working for researchers and developers

This post was compiled by Mark Woodbridge, Jeremy Cohen and Tony Yang of Imperial College’s Research Software Community.

As COVID-19 drives us into uncharted territory, many of us at Imperial will be having our first ever experience of working off-campus for an extended period of time. It, of course, depends on our role, but many members of the College community will be no stranger to mobile working – pitching up at one of the many campus cafes, breakout spaces or a coffee shop, getting out our laptop or mobile device and switching very quickly into a state of focused work. Maybe finishing those next couple of paragraphs of a paper or report, fixing that annoying bug in our scientific code that someone just reported, or responding to an urgent technical query from a collaborator. Sometimes a change of space or environment provides just that little shift in perspective that you need to help solve that challenging technical problem, or get the right wording for that difficult section of the paper, much more quickly than if you’d sat in your office staring at your screen for hours!

Over the coming weeks, we’ll be facing a rather different reality of remote working which is likely to involve spending a significant amount of time working in one space, without the flexibility that comes from being on a large campus. While our primary concern is going to be for the health and safety of our family, friends and colleagues, many of us will also have concerns about how we’ll manage to work effectively in these difficult times. We may have worries about feeling isolated, about maintaining our research efficiency and quality, about meeting deadlines, or more generally about how things will change in our day-to-day working lives as our routines are uprooted completely!

Within Imperial’s Research Software Community, many of us are software developers (Research Software Engineers) or academics/researchers who spend a significant amount of time writing software. The software developer community has embraced remote working over recent years and there are now many examples of companies that operate an entirely remote model with individual developers distributed around the world. If you’re a developer with a laptop and a good internet connection, location is no longer a barrier. In the research community, things are a little different and while many of us will be aware of cases where individuals spend the bulk of their time working remotely, discussion, collaboration and the opportunities posed by ad hoc meetings in the common room make working in a campus environment important and beneficial. Nonetheless, one huge benefit of the wide-ranging use of remote working in the software community is the wealth of tools, advice and examples now out there that make lone, remote working much easier.

A few members of Imperial’s research software community have come together (remotely!) to provide some tips, examples and advice that we hope might be helpful if you’re working remotely. There are many similar articles online but here we’ve tried to provide some thoughts and examples from our own experiences and we hope that these will be particularly relevant to members of the College and its research community. We’ve marked resources only accessible to Imperial members with an asterisk.

1) Communicating with colleagues

Even if you don’t consider yourself to be the most outgoing person, you shouldn’t underestimate the importance of communication with colleagues or collaborators when you’re working alone. If we’re on campus most of the time, we probably have many informal chats with others in our office, people we bump into in the corridor or coffee room, etc.

Think about perhaps scheduling at least one 30 minute catch up with one or two colleagues each day. It doesn’t need to be time wasted through unstructured chat, although even this sort of meeting can be really valuable in helping you to feel connected and ultimately helping to improve your wellbeing. The College recommends and supports the use of Teams but many other solutions are available.

Some other suggestions:

  • Deliberately check in with others and ask how they are – especially if you know they are isolated.
  • Video calling, however uncomfortable to start with, can go some way towards replicating the interactions we’re used to in the campus environment.
  • Celebrate and share achievements, however large or small – from bug-fixes to new releases of your code!
  • Try remote pair programming or debugging: e.g. Live Share
  • Take part in an online community. Our local Research Software Community is on Slack*. There is a new Remote Working Wellbeing* group on Yammer. Outside Imperial many Meetup groups and other events (e.g. CW20) are now going online.
  • Reach out to others: whether housemates, or your local community via Facebook or other virtual groups. Consider volunteering where it is appropriate to do so.
  • Contribute to an open source project. Open source projects (such as The Turing Way) tend to have an established and inviting online community. If it is complementary to your work, and you have the capacity to do so, then making a contribution – even fixing a typo – can be a very fulfilling experience and introduce you to a broader community.

2) Maximising focus

Some people will be used to working from home for at least one day a week – perhaps in an environment that enables us to concentrate at least as well as in the office. But many of us won’t have anything resembling a home-office (or even a desk!) and may have caring or other responsibilities that are difficult to combine with sustained focused work. Generic advice is therefore almost impossible to provide, but here are some ideas:

    • When working in isolation without scheduled meetings or other engagements it can be easy to confuse time spent working with actual productive hours. Try setting alarms or using a timer for focused periods.
    • Messaging apps are great for keeping in touch but can also provide a stream of interruptions. Decide when you’ll be online and offline and set/indicate your status appropriately. And conversely, be mindful about how and when you contact others.
    • If you’re able to control your hours and environment then take advantage: work when you’re most productive, listen to music for programming, ambient sounds… or simply concentrate in potentially unfamiliar (but welcome!) peace and quiet.
  • Delineate your working day (and your workspace) – consciously decide when you’re working and when you’re not, and somehow communicate this to those around you.
  • You may need to be especially creative if you do have caring responsibilities. Don’t be afraid to adopt a working pattern or shifts different to those of your colleagues – as long everyone is aware and can continue to communicate effectively. You may find these Parent Scheme resources helpful.
  • Take breaks, rehydrate, try to eat healthily (especially considering the reduced physical activity you may be getting), and try to get some fresh air. Imperial has a virtual running club if you’re looking for some motivation!
  • Take advantage of the time saved by not commuting: perhaps by taking up a new hobby – ideally something that exercises a different part of your brain! Consider trying meditation: the College Chaplaincy is offering remote sessions and there is also a Mindfulness group* on Yammer.

3) Working comfortably

Without a home office and the availability of the usual alternatives such as libraries, shared workspaces or even coffee shops it can be difficult to find a comfortable place to work for prolonged periods. Ideally find more than one place where you can work and then alternate – even if one is the sofa! Experiment: improvise a standing desk (maybe putting your stockpile to good use…). Take breaks to relieve any tension and give your body a break by stretching or trying some beginner’s yoga. However, if you feel that your health and/or productivity is affected then don’t hesitate to talk to your supervisor or to Occupational Health, who have published some tips for remote working.

A new way of working

We hope that some of these ideas can ease the transition that will undoubtedly be challenging for some of us. But it’s also an opportunity to reassess how we work and how it fits around the rest of our lives. So try to establish clear boundaries between work and relaxation time and spaces, make yourself comfortable, and connect with colleagues, friends and family where you can. Also remember to take enough time off and do not work for prolonged periods without breaks in order to avoid burnout. Transitioning into remote working is a process and a reduction in productivity initially can be expected to happen. Aim to develop a routine, but in the meantime be patient and experiment. Don’t worry, you will soon learn how you work most productively, and hopefully pick up some good habits for the longer term! But if you do struggle then be sure to communicate, take advantage of the many resources out there that can provide help, and ask for advice and assistance if necessary.

Keep safe and we wish you lots of productive (remote) coding, paper writing or research!

Further reading

Did we miss any useful resources? Join the discussion on Slack* or let us know @ImperialRSE!

* As a post targeted primarily at members of the Imperial College London community, this article includes some links that will be accessible only to members of the College. These are marked with an asterisk. Nonetheless, we have included many publicly accessible links and if you are not a member of the College community, we hope you’ve found the content interesting and helpful.

 

The content of this blog post is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) licence.

Running Jupyter notebooks on Imperial College’s compute cluster

We were really glad to see James Howard (NHLI, Faculty of Medicine) announcing on Twitter that he’d published a Kaggle kernel to accompany his recent publication on MR image analysis for cardiac pacemaker identification using neural networks via PyTorch and torchvision. Sharing code in this way is a great way to promote open research, enable reproducibility and encourage re-use.

Figure 3 from Cardiac Rhythm Device Identification Using Neural Networks

We thought it might be helpful to explain how to run similar notebooks on Imperial’s cluster compute service, given that it can provide some benefits while you’re developing code:

  • Your code and data remain securely on-premise, thanks to the RCS Jupyter Service and Research Data Store
  • You can run parallel interactive and non-interactive jobs that span several days, across multiple GPUs

With James’ permission we’ve lightly modified his notebook and published it in an exemplar repository alongside some instructions to run it on the compute cluster. We hope this can help others to use a combination of Conda, Jupyter and PBS in order to conduct GPU-accelerated machine learning on infrastructure managed by the College’s Research Computing Service – without incurring any cost at the point of use.

Many thanks to James Howard for sharing his notebook and reviewing our instructions

RSLondonSouthEast 2020

RSLondonSouthEast 2020, the annual gathering for Research Software Engineers based in or around London, took place on the 6th February at the Royal Society. The College was strongly represented by contributions from RSEs based at Imperial.

Full talks:

Lightning talks:

Posters:

Jeremy Cohen introduces RSLondonSouthEast 2020 at the Royal Society

Jeremy Cohen (Department of Computing) was the chair of the organising committee. Stefano Galvan (Department of Mechanical Engineering), Alex Hill (Department of Infectious Disease Epidemiology) and Jazz Mack Smith (Department of Metabolism, Digestion and Reproduction) served on the programme committee.

Many thanks to all the committee members and everyone who presented, submitted proposals or attended on the day, and to EPSRC and the Society of Research Software Engineering for their support. For more information from the event check Jeremy’s full report, RESIDE’s blog post or #RSLondonSE2020 on Twitter.

Quilting with Julia, or how to combine parallelism and derived types for high performance computing

Research and quilting have a similar Zen in that both combine and build upon multiple prior works. But the workflow is difficult to reproduce in research software: how can we combine group X’s state-of-the-art ODE solver with group Z’s state-of-the-art parallel linear algebra to create Y’s new biology model when they all use different libraries and conventions? This is the problem that Julia tackles head on, thanks to it’s innovative type system and multiple dispatch. In “Shared Memory Parallelization of Banded Block-Banded Matrices” we describe how to combine the parallelization capabilities from one package (SharedArrays) with the specialized matrix  of another (BlockBandedMatrices.jl) – without modifying the internals of either.

This work follows on from a NumFOCUS sponsored collaboration at Imperial College between the Research Computing Service and Sheehan Olver in the Department of Mathematics.

A review of the RSE team’s activities in 2019

2019 has been another very busy and productive year for the RSE team in the Research Computing Service at Imperial College. Our core mission is to accelerate the research conducted at Imperial through collaborative software development, and we have now completed 24 projects since our inception 2 years ago with 75% of our first-year projects resulting in follow-on engagements. We’ve highlighted 5 of our most fruitful collaborations on our new webpages, which also provide more information about the team and the services we offer. We are about to appoint our fifth team member, reflecting the value we’ve offered to research projects (and proving that there is a career pathway for RSEs!).

In addition to our project work we’ve assisted researchers at over 40 RCS clinics this year and played a strong supporting role in Imperial’s Research Software community, from Hacktoberfest to departmental events. We’ve developed two brand new Graduate School courses in Research Software Engineering (to be delivered next term) and have helped deliver 4 Software Carpentry workshops. We’ve also played an increasingly active role in promoting the benefits of RSE (and the role itself) to relevant stakeholders in the College. This has complemented our broader engagement activities: acting as expert reviewers for JOSS submissions, contributing to numerous OSS projects, presenting at 3 international RSE conferences (deRSE19, UKRSE19 and NL-RSE19), and promoting our work via blogging, social media and attendance at several other relevant events – locally (e.g. RSLondonSouthEast 2019) and nationally (e.g. CW19, CIUK).

RSE19 conference photograph
The team (amongst amongst many other RSEs!) at UKRSE19. Photo courtesy @RSEConUK.

We continue to develop tools and infrastructure to support RSE within in the College. The nascent Research Software Directory aims to showcase the breadth of software developed at Imperial, encouraging collaboration, re-use and citation. We’re also attempting to give software a stronger position amongst research outputs through our current work on the Research References Tracking Tool (R2T2) and helping researchers submit their software to Spiral via Symplectic. Finally, we continue to share advice and guidance on how to adopt better RSE practices, such as QA and CI.

As we look forward and further develop the Research Computing Service’s RSE capacity and expertise we’d like to thank all the academics who have trusted us with their projects, and all the researchers who’ve taken the time to explain their work and have enthusiastically embraced good software engineering practices. We’re looking forward to another 12 months of strengthening RSE at Imperial!

1st Research Software Winter Seminars and Roundtable

On Thursday 12th of December the Research Computing Service joined the College’s Research Software Community in celebrating the 1st Research Software Winter Seminars and Roundtable, the final event of another great year of building research software at Imperial. The event had two goals: first, to celebrate the research software-related achievements of the RS Community during 2019, and second, to plan the activities and goals for the year that is about to start.

The seminar session featured nine exciting talks, ranging from a review of the activities of the Community during 2019 and the training opportunities in computing and data science skills, to technical talks on the use of complex analysis pipelines for RNA sequencing and the extension of open source software with custom features.

This is the full list of talks, including several relevant links:

After the talks, there was a roundtable discussion chaired by Diego Alonso, with a panel including Elsa Angelini, Jeremy Cohen, Phoebe Pearce and Mark Woodbridge, to help answer some questions about what the audience would like to see from the Community next year, how we can communicate with each other better and who can get involved to make those things happen. There were many excellent contributions from the audience, who were also very engaged and eager to see the community grow and take an active role on it.

Among the activities that were discussed – and that gained volunteers to help make them a reality – were the creation of a Slack workspace as an instantaneous, bidirectional communication channel within the community (already up and running; sign-up now!) and the recruitment of RSE Champions in the different communities (PhD students, postdocs, etc) to promote Community events and bring more people aboard or to assist with the organisation of departmental events.

The event concluded with informal drinks and nibbles in the ICT Kitchen – including mulled wine! – where the enthusiastic attendees and speakers mingled together and shared experiences and plans for the future.

There are plenty of things going on and 2020 is due to see a very bright RS Community at Imperial!

NL-RSE19

On 20 November 2019 Mark Woodbridge and Jeremy Cohen represented Imperial College at NL-RSE19, the first annual conference of the Netherlands Research Software Engineer community.

NL-RSE19 poster session

Their presentation, Strength in Numbers: Growing RSE Capacity at Imperial College London (10.5281/zenodo.3548308) described the expanding groups involved in RSE at Imperial, their respective activities, and how examples of these are fostering collaboration and awareness across the College. They also took the opportunity to display a poster first shown at UKRSE19 that highlights key aspects of these initiatives. The talk and poster generated much interest and resulted in productive discussions with members of the NL-RSE community in relation to building inclusive communities, long-term support for research software, personal development opportunities for RSEs, and how best to support the broad range of research typically carried out in larger institutions.

NL-RSE19 poster session

Many thanks to the organisers (in particular Niels Drost and Ben van Werkhoven of the Netherlands eScience Center) for the opportunity to engage with the vibrant and rapidly growing RSE community in the Netherlands.